CN114663292A - Ultra-lightweight picture defogging and identification network model and picture defogging and identification method - Google Patents

Abstract

The application discloses an ultra-lightweight picture defogging and identification network model, through which the defogging and identification of pictures are realized, and the network model comprises a bidirectional GAN network model and a target detection network model which are sequentially connected. And the bidirectional GAN network model demists the fog image and outputs a clear image to the target detection network model for feature recognition processing. And performing pruning retraining on the target detection network model, wherein the training process comprises the steps of training the original images of the training set for multiple times, performing down-sampling on the original images by preset times before each training, sequencing and comparing the scaling coefficients of the batch normalization layer after each training, and removing the previous layer of convolution kernels corresponding to the channels of which the scaling coefficients are smaller than the preset scaling threshold value to realize pruning. The target detection network model is further pruned on the basis of the existing micro recognition model, the scale of ultra-lightweight picture defogging and the recognition network model is greatly reduced, and the target detection network model can be deployed on an end-side platform with limited calculation capacity and power consumption resources.

Description

Ultra-lightweight image dehazing and identification network model, image dehazing and identification method

technical field

The present application relates to the technical field of image processing, and in particular, to an ultra-lightweight image dehazing and identification network model, and an image dehazing and identification method.

Background technique

The hazy weather will reduce the clarity of the pictures captured by the camera, making it difficult for the computer to recognize the features of the objects in the images. Therefore, before the image is recognized, the image needs to be dehazed. At present, the dehazing and recognition processing of pictures can be achieved through neural network models.

For dehazing processing, the commonly used neural network model is Generative Adversarial Networks (GAN), which adopts an unsupervised learning method to learn the mapping relationship between foggy pictures and clear pictures through a large amount of training data, and then Dehaze the fog map. For recognition processing, the commonly used neural network model is the YOLO (You Only Look Once) target detection model. The network model divides the input image into S*S grids, and each grid is responsible for detecting objects whose center falls within itself. By directly regressing the regression frame of the target and predicting the category at multiple positions of the input image, the recognition of the feature of the picture object is realized.

The current YOLO target detection model is relatively large, and the neural network model obtained in the process of applying it to image dehazing and recognition is not suitable for deployment on end-side platforms with limited computing power and power consumption resources, such as mobile phones or autonomous driving related on the device.

SUMMARY OF THE INVENTION

In order to solve the problem that the current YOLO target detection model is relatively large, the neural network model obtained in the process of applying it to image dehazing and recognition is not suitable for deployment on end-side platforms with limited computing power and power consumption resources. The following embodiments disclose ultra-lightweight image dehazing and identification network models, and image dehazing and identification methods.

A first aspect of the present application discloses an ultra-lightweight image dehazing and recognition network model, including: a bidirectional GAN network model and a target detection network model connected in sequence;

The two-way GAN network model is used to process the input image to be dehazed and output a clear image, and the target detection network model is used to perform feature recognition processing on the clear image;

The target detection network model is a Yolo-Tiny-S network model that has undergone row pruning and retraining; in the row pruning and retraining process, the original images in the training set of the target detection network model are trained multiple times, and each training Before, the original image is down-sampled by a preset multiple, and after each training is completed, the scaling coefficients of the batch normalization layer are sorted and compared, and the previous layer corresponding to the channel whose scaling coefficient is smaller than the preset scaling threshold is sorted and compared. The convolution kernel is removed to realize pruning;

The target detection network model includes a feature direct processing module and a feature fusion processing module. The feature direct processing module includes a front extraction unit, a middle extraction unit and a first rear extraction unit that are connected in sequence. The front extraction unit is input into the target detection network model, the feature fusion processing module includes a feature fusion splicing unit and a second rear extraction unit that are connected in sequence, and the outputs of the front extraction unit and the middle extraction unit are The terminals are all connected to the input terminal of the feature fusion splicing unit.

Optionally, the front extraction unit includes a DBL-S combination subunit and a plurality of MDBL-S combination subunits connected in turn;

The middle extraction unit includes a plurality of the MDBL-S combination subunits and one of the DBL-S combination subunits connected in sequence;

The first rear extraction unit and the second rear extraction unit include a DBL-S combination subunit and a convolution subunit connected in sequence;

The feature fusion and splicing unit includes an upsampling subunit and a feature splicing subunit, the input end of the upsampling subunit is connected with the output end of the middle extraction unit, and the output end of the upsampling subunit is connected to the The output end of the front extraction unit is connected to the input end of the feature splicing subunit;

The DBL-S combination subunit is composed of a dark network convolution layer, a batch normalization layer and a leakage correction linear layer connected in sequence;

The MDBL-S combination subunit is composed of the sequentially connected max pooling layer and the DBL-S combination subunit.

Optionally, the bidirectional GAN network model includes an input module, a generation module and a discrimination module;

The input module includes a clear image input port and a fog image input port, the generation module includes a first generation unit and a second generation unit, and the discrimination module includes a first discriminator and a second discriminator;

The clear image input port, the first generating unit and the first discriminator are connected in sequence for feature extraction and reconstruction for the clear image, the fog image input port, the second generating unit and the The second discriminators are connected in sequence, and are used for feature extraction and reconstruction for the fog map;

The first generation unit includes a first encoder, a shared latent space, and a first decoder connected in sequence, and the second generation unit includes a second encoder, the shared latent space, and a second decoder connected in sequence;

The shared latent space is used for storing high-level features, and outputting the high-level features to the first decoder and the second decoder, the high-level features including the high-level features extracted by the first encoder for sharp images features and high-level features extracted by the second encoder for the haze image;

Both the training set and the verification set of the bidirectional GAN network model include pairs of clear images and fog images.

Optionally, the first encoder and the second encoder respectively include a first convolution block, a second convolution block and a first coupled residual block connected in sequence;

The first decoder and the second decoder respectively include a second coupled residual block, a first-sized convolutional block and a second-sized convolutional block connected in sequence;

The output of the first coupled residual block is connected to the input of the shared latent space, and the output of the shared latent space is connected to the input of the second coupled residual block;

The output terminal of the first convolution block is jump connected to the input terminal of the second step size convolution block;

The output terminal of the second convolution block is jump connected to the input terminal of the first length convolution block.

Optionally, the first coupled residual block and the second coupled residual block are respectively formed by concatenating multiple sub-residual blocks;

The sub-residual block at any stage includes a first convolution layer, an activation function layer and a second convolution layer that are connected in sequence, and are used to process the output results of the sub-residual block of the previous stage and the sub-residual of the previous stage. The output result of the block, and the processing result is output to the next-level sub-residual block and the next-level sub-residual block.

Optionally, both the first discriminator and the second discriminator include three layers of discriminant networks and one layer of granularity discriminant networks, and any layer of the discriminant network includes three convolution layers and one activation function layer.

Optionally, the bidirectional GAN network model is optimized by the following loss functions: a generative adversarial loss function, an MSE loss function, and a total variational loss function.

A second aspect of the present application discloses a method for image dehazing and identification, the method comprising:

Extract the depth information corresponding to different pixels in the image to be dehazed;

performing depth processing on the picture to be dehazed according to the depth information;

Input the deeply processed image to be dehazed into a pre-built ultra-lightweight image dehazing and recognition network model;

Obtain the image dehazing and recognition results output by the ultra-lightweight image dehazing and recognition network model.

Optionally, the extracting depth information corresponding to different pixels in the image to be dehazed includes:

Perform format conversion on the picture to be dehazed, and extract the brightness and saturation of the picture to be dehazed;

According to the brightness and saturation, depth information corresponding to different pixels in the image to be dehazed is generated.

A third aspect of the present application discloses a computer device, comprising:

memory for storing computer programs;

The processor is configured to implement the steps of the image dehazing and identification method according to the second aspect of the present application when executing the computer program.

A fourth aspect of the present application discloses a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is processed and executed, the image dehazing and Identify the steps of the method.

The present application discloses an ultra-lightweight image dehazing and recognition network model, and realizes image dehazing and recognition through the network model. The network model consists of a bidirectional GAN network model and a target detection network model that are connected in sequence. The bidirectional GAN network model is used to process the input image to be dehazed and output a clear image, and the target detection network model is used to perform feature recognition processing on the clear image. The target detection network model is the Yolo-Tiny-S network model that has undergone row pruning and retraining. During the row pruning and retraining process, the original images in the training set of the target detection network model are trained multiple times. The image is down-sampled by a preset multiple. After each training is completed, the scaling coefficients of the batch normalization layer are sorted and compared, and the convolution kernel of the previous layer corresponding to the channel whose scaling coefficient is less than the preset scaling threshold is removed to realize Pruning. The target detection network model disclosed in this application is further pruned on the basis of the current micro-recognition model Yolov3-tiny, which greatly reduces the scale of the ultra-lightweight image dehazing and recognition network model, and can be easily deployed in computing power It is more practical to complete high-speed target detection by integrating it into mobile-end chips such as end-side platforms or vehicle-mounted cameras with limited power consumption resources.

Description of drawings

In order to illustrate the technical solutions of the present application more clearly, the accompanying drawings required in the embodiments will be briefly introduced below. Obviously, for those of ordinary skill in the art, without creative work, the Additional drawings can be obtained from these drawings.

1 is a schematic structural diagram of an ultra-lightweight image dehazing and identification network model disclosed in an embodiment of the present application;

2 is a schematic structural diagram of a target detection network model disclosed in an embodiment of the present application;

FIG. 3 is another schematic structural diagram of a target detection network model disclosed in an embodiment of the present application;

4 is a schematic structural diagram of an existing dehazing convolutional neural network;

5 is a schematic structural diagram of a bidirectional GAN network model disclosed in an embodiment of the application;

6 is a schematic structural diagram of another bidirectional GAN network model disclosed in an embodiment of the present application;

7 is a schematic diagram of a shared latent space in a bidirectional GAN network model disclosed in an embodiment of the present application;

8 is a schematic structural diagram of a first generation unit and a second generation unit in a bidirectional GAN network model disclosed in an embodiment of the present application;

9 is a schematic structural diagram of a first coupled residual block and a second coupled residual block in a bidirectional GAN network model disclosed in an embodiment of the present application;

10 is a schematic structural diagram of a first discriminator and a second discriminator in a bidirectional GAN network model disclosed in an embodiment of the application;

11 is a schematic diagram of calculating cross-domain transformation consistency in a bidirectional GAN network model disclosed in an embodiment of the application;

12 is a schematic diagram of the application of a bidirectional GAN network model disclosed in an embodiment of the application;

FIG. 13 is a schematic workflow diagram of a method for image dehazing and identification disclosed in an embodiment of the present application.

Detailed ways

In order to solve the problem that the current YOLO target detection model is relatively large, the neural network model obtained in the process of applying it to image dehazing and recognition is not suitable for deployment on end-side platforms with limited computing power and power consumption resources. The following embodiments disclose ultra-lightweight image dehazing and identification network models, and image dehazing and identification methods.

Referring to FIG. 1 , an ultra-lightweight image dehazing and recognition network model disclosed in the first embodiment of the present application includes: a bidirectional GAN network model and a target detection network model that are connected in sequence.

The two-way GAN network model is used to process the input image to be dehazed, and output a clear image, that is, a dehazed image.

The target detection network model is used to perform feature recognition processing on the clear image. The target detection network model is a Yolo-Tiny-S network model retrained by row pruning. In the process of row pruning and retraining, the original images in the target detection network model training set are trained for multiple times. Before each training, the original images are down-sampled by a preset multiple. The scaling coefficients of the batch normalization layer are sorted and compared, and the convolution kernel of the previous layer corresponding to the channel whose scaling coefficient is smaller than the preset scaling threshold is removed to realize pruning.

In this embodiment, in order to enable the target detection network model to better complete rapid detection, the size of the model is compressed and the size of the input image is reduced, but the convergence performance of the model needs to be guaranteed during the compression process, so the row pruning and retraining method is adopted. Model training, the specific process includes:

The first step is to directly train the original images in the training set. At this time, the resolution of the original images is larger, and data enhancement is used during the training process, and a larger scaling ratio is used to create training images of different sizes.

The second step is to sort and compare the scaling coefficients of the batch normalization layer after training, and prune the channels with smaller scaling coefficients and compression requirements. Specifically, the convolution kernels of the previous layer corresponding to the channels with smaller scaling coefficients are removed, and only the convolution kernels corresponding to the channels with larger scaling coefficients in the batch normalization layer are retained.

The third step is to downsample the original image by a factor of two, and repeat the first and second steps to train again. During the training process, reduce the zoom ratio to perform image enhancement on the original image, so as to avoid the low resolution of some objects during the zooming process of the small image, which will affect the convergence of the model. It should be noted that in the process of sorting and comparing the pruning channels of the batch normalization layer, the proportion of pruning channels is increased, and the channel is further reduced.

The fourth step is to downsample the original image by a higher multiple, and repeat the first and second steps for retraining until the final calculation amount and input image size meet the requirements of resources and speed. For example, the original image resolution is reduced from 512*512 to 128*128 at the time of input, and the number of model channels is reduced to about half of the current general ultra-lightweight target detection model Yolov3-tiny.

When the model converges to a better accuracy, connect the trained ultra-lightweight target detection model Yolo-Tiny-S to the output port of the bidirectional GAN network model to form an integrated model of dehazing and recognition, that is, ultra-lightweight A high-level end-to-end image dehazing and recognition network model, through which the target recognition can be performed directly on the existing real fog images in an end-to-end manner.

In this embodiment, an ultra-lightweight target detection model Yolo-Tiny-S (miniature Yolo detection model) is proposed for the first time, and the two-way GAN network model is combined to complete target detection. The ultra-lightweight target detection model is a lightweight target detection model. The target detection model Yolov3-tiny is based on batch normalization layer, which is trained for iterative channel high-proportion pruning of small input images. This training scheme is first proposed for this scheme, and the scale of the obtained target detection network model is much smaller than that of predecessors. The Faster-RCNN used in the system, the pruning iterative algorithm can effectively converge the network that can quickly identify small images, which is beneficial to the real-time application of the model in practical application scenarios, such as the Internet of Vehicles, mobile phones or autonomous driving and other resource-constrained platforms. Target Detection.

Referring to FIG. 2, the target detection network model includes a feature direct processing module and a feature fusion processing module. The feature direct processing module includes a front extraction unit, a middle extraction unit and a first rear extraction unit that are connected in sequence. The clear A graph is input into the target detection network model through the front extraction unit. The feature fusion processing module includes a feature fusion splicing unit and a second rear extraction unit that are connected in sequence, and the output ends of the front extraction unit and the middle extraction unit are both connected to the input end of the feature fusion splicing unit.

Referring to Fig. 3, the feature fusion splicing unit includes an upsampling subunit and a feature splicing subunit, the input end of the upsampling subunit is connected with the output end of the middle extraction unit, and the upsampling subunit's output end is connected. The output end and the output end of the front extraction unit are connected to the input end of the feature splicing subunit.

The front extraction unit includes a DBL-S combination sub-unit and a plurality of MDBL-S combination sub-units connected in sequence. In this embodiment, the front extraction unit includes one DBL-S combination subunit and four MDBL-S combination subunits that are connected in sequence.

The middle extraction unit includes a plurality of the MDBL-S combination subunits and one of the DBL-S combination subunits connected in sequence. In this embodiment, the front extraction unit includes two MDBL-S combining subunits and one DBL-S combining subunit which are connected in sequence.

Both the first post extraction unit and the second post extraction unit include one of the DBL-S combining subunits and one convolution (Conv) subunit connected in sequence.

The DBL-S combination subunit is composed of a dark network convolutional layer (DarkNet convolutional layer), a batch normalization layer (BN layer) and a leak correction linear layer (LeakyRelu layer) connected in sequence. Among them, S refers to Slim, which is used to indicate that the corresponding module has been reduced, and the dark network convolution layer is a channel-reduced dark network.

The MDBL-S combination subunit is composed of a maximum pooling layer (MaxPool layer) and the DBL-S combination subunit connected in sequence.

Combined with Figure 2 and Figure 3, the clear image after dehazing is used as input, and the feature extraction is completed through the combination of the front extraction unit and the middle extraction unit, and then two different feature maps of the same size are obtained from the middle and rear of the network for prediction. . Specifically, each grid on a feature map predicts 3 prediction frames, and each prediction frame requires (x, y, w, h, confidence) (predicted x position, predicted y position, predicted frame width, predicted frame Height, prediction confidence) five basic parameters, indicating the center point position, width and height of the prediction frame, and the confidence of the existence of the frame, and output a probability for each category. After that, use the feature splicing subunit to upsample the feature map drawn from the back of the network, and connect it with the feature map in the middle of the network (ie, the output of the front extraction unit) to form a new fusion feature map. The feature map and the new fusion feature map directly generated at the back of the network (ie, the output of the middle extraction unit) are sent to the first and second rear extraction units, respectively, to generate the final prediction classification and regression frame, The clear picture after dehazing can be recognized. In Figure 3, y1 and y2 represent two branches with the same function. Each of y1 and y2 contains the information of the predicted classification result and the regression frame. The predicted classification result refers to the image detected by the image. The object is what type of object is identified, and the regression box is the box that represents the position of the detected object.

For image dehazing, the commonly used dehazing convolutional neural network is Generative Adversarial Networks (GAN). Produces highly accurate output. The generation module includes an encoder and a decoder. As shown in Figure 4, the encoder is used to extract the feature vector in the input picture, and the decoder is used to restore the low-level features from the feature vector.

However, in the prior art, only one-way conversion from fog map to clear map is performed through GAN, so GAN only includes the one-way mapping relationship from fog domain to clear domain. After the fog map is processed by the encoder and decoder, the output There are usually halo effects and artificial artifacts in the clear images of the original image, and the information of the original image is not well preserved.

This embodiment involves a bidirectional GAN network model, see FIG. 5 , the network model includes an input module, a generation module and a discrimination module.

The input module includes a clear image input port and a fog image input port, the generation module includes a first generation unit and a second generation unit, and the discrimination module includes a first discriminator and a second discriminator.

The clear image input port, the first generating unit and the first discriminator are connected in sequence for feature extraction and reconstruction for the clear image, the fog image input port, the second generating unit and the The second discriminators are connected in sequence, and are used for feature extraction and reconstruction for the fog image.

The two-way GAN network model shown in Figure 5 can dehaze the fog image, and can also reverse the fog on the clear image to enhance the consistency of changes across domains, and thus better constrain the mutual The conversion effect makes the dehazing image more natural when applied, and enhances the dehazing effect on the real fog map. However, in the current dehazing technology, only GAN is used to do the one-way conversion from the fog image to the clear image, and only the mapping relationship between the fog domain and the clear domain can be output, so that the output of the image has halo effect and artificial defects. The original image information is not well preserved.

Referring to FIG. 6 , in the bidirectional GAN network model designed in this embodiment, the first generation unit includes a first encoder, a shared latent space, and a first decoder connected in sequence, and the second generation unit includes a sequentially connected first encoder. Two encoders, the shared latent space and a second decoder.

The shared latent space is used for storing high-level features, and outputting the high-level features to the first decoder and the second decoder, the high-level features including the high-level features extracted by the first encoder for sharp images features and high-level features extracted by the second encoder for the haze image.

Both the training set and the verification set of the bidirectional GAN network model include pairs of clear images and fog images.

In this embodiment, it is assumed that a latent space is shared between the fog map and the clear map, and the feature connection from the fog map to the clear map can be completed through the latent space. Referring to Fig. 7, clear image X and fog image Y, which can be connected by a shared latent space Z, recover images in both domains. Based on this theoretical basis, the bidirectional GAN network model disclosed in this embodiment can process real fog images and synthetic fog images very well, and recover the fog images by combining the bidirectional generative adversarial network with the method of encoding first and then decoding. reconstruction. The two-way GAN network model completes training and verification through paired clear images and fog images, including the two-way mapping relationship between the fog domain and the clear domain, and can process images in different domains, effectively ensuring the authenticity of image reconstruction.

This embodiment discloses a bidirectional GAN network model as a dehazing network to effectively deal with real dense fog. The network structure is novel and effective, and it can extract the depth information of fog images at different depths. It is a great advantage, which can greatly improve the accuracy of subsequent object recognition. Due to the limitation of the dehazing model in previous methods, there is no way to directly train with real haze images, so the corresponding dehazing results cannot be well adapted to the detection in the actual haze scene. The two-way GAN network model can use real images for end-to-end training, so that the information of real fog images can be better used to assist the improvement of detection effects in real fog image scenes. Through joint training of real and synthetic data sets, it can be more Make good use of the fog map information in the real scene to avoid the difficulty of network training caused by the lack of paired training data. The learning of real information enables the subsequent target detection network model to achieve good generalization effects in real scenes. .

Further, as shown in FIG. 8 , the first encoder and the second encoder respectively include a first convolution block, a second convolution block and a first coupled residual block which are connected in sequence.

The first decoder and the second decoder respectively include a second coupled residual block, a first-sized convolutional block, and a second-sized convolutional block connected in sequence.

The output of the first coupled residual block is connected to the input of the shared latent space, and the output of the shared latent space is connected to the input of the second coupled residual block.

The output of the first convolution block is jump connected to the input of the second stride convolution block.

The output terminal of the second convolution block is jump connected to the input terminal of the first length convolution block.

Among them, the first convolution block and the second convolution block are used to extract high-level features of the input picture, and the first coupled residual block and the second coupled residual block are responsible for learning the detailed information of different features of the picture, so as to facilitate the restoration and reconstruction of the picture. The first step convolution and the second step convolution complete the reconstruction process of the image from high-level features to the output image (clear image or fog image). The encoder and decoder are tightly connected by jump connections. After each layer of convolution block, there will be a jump link linked to the corresponding stride convolution block. Before the jump link, the output feature map of the encoder's convolution block is linked to the input of the corresponding decoder. The splicing is performed at the dimension level, so that the feature map output by the decoder can include the feature information of the original image. The jump connection enables the detailed texture of the picture to be better learned. Through the connection of two layers of jump connections in high and low dimensions, the different position information of the picture can be processed separately, which further ensures that the output picture and The consistency of the input image, and the authenticity of the reconstruction.

Further, referring to FIG. 9 , the first coupled residual block and the second coupled residual block are respectively formed by concatenating multiple sub-residual blocks.

The sub-residual block at any stage includes a first convolution layer, an activation function layer and a second convolution layer that are connected in sequence, and are used to process the output results of the sub-residual block of the previous stage and the sub-residual of the previous stage. The output result of the block, and the processing result is output to the next-level sub-residual block and the next-level sub-residual block.

The coupled residual block can enhance the ability of the network to process information and better fit the potential identity mapping of the deep network. In this embodiment, each sub-residual block is composed of two convolutional layers and one activation function layer. (Fig. 9 shows that the outputs and inputs of the three) sub-residual blocks are cascaded with each other, and each sub-residual block processes the input of the current stage and the low-dimensional input results of the previous stage respectively to extract features of different dimensions information, and send the calculation results to the next-level sub-residual block and the next-level sub-residual block respectively. Through the tight connection of multi-layer sub-residual blocks, the residual information of fog features of different dimensions can be learned.

After the encoding and decoding of the encoder and the decoder, the generated picture is obtained, and the generated picture is input to the discrimination module for discrimination. Referring to FIG. 10 , the first discriminator and the second discriminator respectively include three-layer discriminant networks, and the discriminant network at any layer includes three convolutional layers and a layer of activation functions, and each layer of the discriminant network is on the top The first-level input is down-sampled, and the output results of different granularities of the three-layer discriminant network are compared with the real feature maps to determine whether the network output meets expectations, impose a penalty term on the training results that do not meet expectations, calculate the discriminant loss, and return it to the network. Update parameters. Using a multi-level discriminant network, the output results generated by the bidirectional GAN network model are sampled and compared with three layers of features, and the coarse-grained and fine-grained comparisons are carried out, which expands the discrimination accuracy of the discriminant module.

For the bidirectional GAN network model, a loss function is used in this embodiment, and the auxiliary model can be better expressed. The optimized loss function includes a generative adversarial loss function, an MSE loss function, and a total variational loss function.

In order to achieve a good training effect, the network needs the loss function to contain a generative adversarial loss, and the generative adversarial loss comes from whether the image comes from the generator G(I) or the real data J. The classic generative adversarial loss is defined as follows:

L _GAN (G, D)=E _I (log D(J))+E _I (log(1-D(G(I)));

Combined with Figure 11, the bidirectional GAN network model generates the adversarial loss in both directions, and the cross-domain transformation consistency is calculated as follows:

_{La adv} =L _GAN (G _c (I _c ), Dis _h )+L _GAN (G _h (I _h ), _Disc );

Among them, LGAN() represents the classical adversarial generation loss, Ih represents the input fog image X, Ic represents the input clear image Y, Gc() corresponds to the generation network in the clear scene, and Gh() represents the corresponding generation network in the foggy scene. , Disc represents the discrimination module in clear scenes, and Dish represents the discrimination module in foggy scenes. By sharing the latent space theory, combined with the bidirectional generative adversarial loss, the authenticity of the reconstructed image and the similarity of the results of cross-domain image transformation can be guaranteed.

In the training of the synthetic fog image, the MSE loss function is used to ensure that the predicted image G(I) is similar to the ground truth J. The MSE loss function looks like this:

L _MSE =||G _c (I _c )-J _h || ₁ +||G _h (I _h )-J _c || ₁ ;

Among them, Jh and Jc represent the fog map data and clear map data in the real scene, respectively, and the loss is determined as the difference between the pixels of the fog (clear) map and the real fog (clear) map ||()||1 represents the image on the image The mean of the squared sum of the errors for the corresponding pixel point (mean squared error).

The total variational loss function is used in the training of the fog image to eliminate the related artificial defects, make the image visual effect better, and preserve the texture and details of the image. The total variational loss function is as follows:

和

表示水平和垂直的梯度差。in,

and

Represents the horizontal and vertical gradient difference.

Through the above training, when the model converges to a better accuracy, the obtained training weights can be saved, and then the final two-way GAN dehazing network model can be obtained. Figure to dehaze.

Referring to FIG. 12 , when the bidirectional GAN network model disclosed in this embodiment is deployed in an actual application scenario, the brief steps for dehazing are as follows:

1. Input the real fog map in the real scene.

2. Preprocess the image, convert from RGB space to HSV space, and combine known parameters to output the depth information of the image.

3. Combine the depth information, process the fog map, decode the fog map and store it in the shared latent space, encode the pictures in the shared latent space, and generate clear and dehazed pictures in the real scene.

In order to enhance the consistency of cross-domain transformation and obtain more realistic dehazing images, the existing one-way GAN network is improved. Based on the same shared latent space, clear map→latent space→fog map and fog map→latent Space→Clear Map Two one-way GAN networks are combined in one module through two sets of encoders and decoders, forming a two-way GAN network model that can generate fog/clear map, and calculate fog map through corresponding discriminator→clear The cross-domain transformation loss of graph→fog graph and clear graph→fog graph→clear graph can better learn the distribution of fog and clear graph.

The bidirectional GAN network model disclosed in this embodiment, based on the shared latent space assumption, proposes a bidirectional dehazing network that encodes first and then decodes. This network can process images in different domains, and is very useful for image conversion across domains. Strong adaptability, capable of processing fog information at different depths, and has strong advantages in remote and dense fog processing. The jump link and coupling residual block and multi-layer discriminant structure are applied, which greatly improves the robustness of the network and can more accurately fit the image distribution of the dehazing clear map. Real fog information can be used, and depth information can be extracted after preprocessing, which is convenient for dealing with fog in deep concentration, and can be well expressed in real scenes.

Using the bidirectional GAN network model disclosed in the above-mentioned embodiment, the image dehazing process can be performed. In practical application, the depth information corresponding to different pixels in the image to be dehazed is firstly extracted, and the depth processing of the image to be dehazed is performed; The deeply processed image to be dehazed is input into the pre-built bidirectional GAN network model, and the dehazed image output by the bidirectional GAN network model can be obtained.

The present application discloses an ultra-lightweight image dehazing and recognition network model, and realizes image dehazing and recognition through the network model. The network model consists of a bidirectional GAN network model and a target detection network model that are connected in sequence. The target detection network model is further pruned on the basis of the current micro-recognition model Yolov3-tiny, which greatly reduces the scale of the ultra-lightweight image dehazing and recognition network model, and can be easily deployed in computing power and power. It is more practical to complete high-speed target detection in terminal-side platforms with limited resource consumption or chips on mobile terminals such as vehicle-mounted cameras.

The ultra-lightweight image dehazing and identification network model disclosed in this application includes two modules: dehazing and identification. The dehazing module adopts a bidirectional GAN network model, and the identification module adopts Yolo-Tiny-S as the target detection network model. Each module can be directly connected on the basis of their independent training to perform target detection in haze scenes with high recognition rate and speed, avoiding the need to mark real haze data sets for target detection tasks during the joint training process. Huge manpower and time consumption. Since the dehazing module has been able to combine the depth information of the real haze map to learn the real distribution of the fog, the connection between the pre-order dehazing module and the object recognition network does not require the target detection data set under the real haze map (this data set It is currently missing) joint training, but the separately trained dehazing module and recognition module can be directly connected to form an integrated working system for dehazing and recognition in real scenes.

Referring to FIG. 13 , the second embodiment of the present application discloses a method for image dehazing and identification, and the method includes:

Step S11, extracting depth information corresponding to different pixels in the image to be dehazed.

Step S12: Perform depth processing on the picture to be dehazed according to the depth information.

Specifically, format conversion is performed on the image to be dehazed, and based on OpenCV, the image to be dehazed is converted from RGB format to HSV format, so as to extract the brightness v and saturation s of the image to be dehazed. Among them, OpenCV is a cross-platform computer vision and machine learning software library based on BSD license (open source). Specifically, in the shadow detection algorithm, the image in RGB format is often converted into HSV format. For the shadow area, its chroma and saturation do not change much compared to the original image, mainly because the brightness information changes greatly. The format is converted to HSV format, and the H, S, and V components can be obtained, thereby obtaining the values of chroma, saturation, and brightness.

According to the brightness v and saturation s, depth information corresponding to different pixels in the image to be dehazed is generated.

In actual operation, according to the brightness v and saturation s, the existing depth information formula is used for calculation, and the known parameters θ0, θ1, θ2 are linearly calculated to obtain the depth information d corresponding to different pixels in the image to be dehazed. .

The depth information calculation formula is as follows:

d(X)=θ0+θ1v(X)+θ2s(X)+ε(X);

Among them, X represents the image to be dehazed, ε(X) is the random variable representing the random error of the model, and ε can be regarded as a random image.

Deep processing is performed on the input original image to be dehazed through depth information to enhance the performance effect of deep areas, such as improving brightness and contrast in distant places, making the distant view clearer, etc.

Step S13: Input the deeply processed image to be dehazed into a pre-built ultra-lightweight image dehazing and recognition network model.

Step S14, obtaining the image dehazing and identification results output by the ultra-lightweight image dehazing and identification network model.

The ultra-lightweight image dehazing and recognition network model disclosed in this application includes two modules: dehazing and recognition. The bidirectional GAN network model of the dehazing module can well extract the depth information of the real haze map and process the synthetic haze map. Through the method of encoding first and then decoding, the picture is interpreted into the shared latent space, and then reconstructed, and then the reconstructed clear picture is sent to the recognition module. The Yolo-Tiny-S model of the recognition module can effectively recognize the object, with To achieve the purpose of object recognition after dehazing.

The third embodiment of the present application discloses a computer device, including:

Memory for storing computer programs.

The processor is configured to implement the steps of the image dehazing and identification method according to the second embodiment of the present application when executing the computer program.

The fourth embodiment of the present application discloses a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is processed and executed, the image removal method described in the second embodiment of the present application is realized. Fog and steps of identification method.

Aiming at the problem of low recognition accuracy of haze images in object recognition tasks, the present application first proposes a network model integrating image dehazing and image recognition for micro and small object recognition networks through the above embodiments. By improving the training model of end-to-end dehazing network, a network model for dehazing and identification is proposed, which is trained by mixing real pictures and synthetic pictures. The model can extract the depth information of pictures for analysis, so that end-to-end dehazing The fog network can correctly analyze the depth details and dense fog information of the image, so that the image can have a better dehazing effect in the dense fog scene, and the model after dehazing can also keep well in the subsequent micro-target detection network. Effect.

The present application has been described in detail above with reference to the specific embodiments and exemplary examples, but these descriptions should not be construed as a limitation on the present application. Those skilled in the art understand that, without departing from the spirit and scope of the present application, various equivalent replacements, modifications or improvements can be made to the technical solutions of the present application and the embodiments thereof, which all fall within the scope of the present application. The scope of protection of the present application is determined by the appended claims.

Claims (10)

1. an ultra-lightweight image dehazing and identification network model, is characterized in that, comprises: two-way GAN network model and target detection network model that are connected successively; The two-way GAN network model is used to process the input image to be dehazed and output a clear image, and the target detection network model is used to perform feature recognition processing on the clear image; The target detection network model is a Yolo-Tiny-S network model that has undergone row pruning and retraining; in the row pruning and retraining process, the original images in the training set of the target detection network model are trained multiple times, and each training Before, the original image is down-sampled by a preset multiple, and after each training is completed, the scaling coefficients of the batch normalization layer are sorted and compared, and the previous layer corresponding to the channel whose scaling coefficient is smaller than the preset scaling threshold is sorted and compared. The convolution kernel is removed to realize pruning; The target detection network model includes a feature direct processing module and a feature fusion processing module. The feature direct processing module includes a front extraction unit, a middle extraction unit and a first rear extraction unit that are connected in sequence. The front extraction unit is input into the target detection network model, the feature fusion processing module includes a feature fusion splicing unit and a second rear extraction unit that are connected in sequence, and the outputs of the front extraction unit and the middle extraction unit are The terminals are all connected to the input terminal of the feature fusion splicing unit. 2. a kind of ultra-lightweight image dehazing and identification network model according to claim 1, is characterized in that, The front extraction unit includes a DBL-S combination subunit and a plurality of MDBL-S combination subunits connected in sequence; The middle extraction unit includes a plurality of the MDBL-S combination subunits and one of the DBL-S combination subunits connected in sequence; The first rear extraction unit and the second rear extraction unit include a DBL-S combination subunit and a convolution subunit connected in turn; The feature fusion and splicing unit includes an upsampling subunit and a feature splicing subunit, the input end of the upsampling subunit is connected with the output end of the middle extraction unit, and the output end of the upsampling subunit is connected to the The output end of the front extraction unit is connected to the input end of the feature splicing subunit; The DBL-S combination subunit is composed of a dark network convolution layer, a batch normalization layer and a leakage correction linear layer that are connected in sequence; The MDBL-S combination subunit is composed of the sequentially connected max pooling layer and the DBL-S combination subunit. 3. a kind of ultra-lightweight image dehazing and identification network model according to claim 1, is characterized in that, described two-way GAN network model comprises input module, generation module and discriminating module; The input module includes a clear image input port and a fog image input port, the generation module includes a first generation unit and a second generation unit, and the discrimination module includes a first discriminator and a second discriminator; The clear image input port, the first generating unit and the first discriminator are connected in sequence for feature extraction and reconstruction for the clear image, the fog image input port, the second generating unit and the The second discriminators are connected in sequence, and are used for feature extraction and reconstruction for the fog map; The first generation unit includes a first encoder, a shared latent space, and a first decoder connected in sequence, and the second generation unit includes a second encoder, the shared latent space, and a second decoder connected in sequence; The shared latent space is used for storing high-level features, and outputting the high-level features to the first decoder and the second decoder, the high-level features including the high-level features extracted by the first encoder for sharp images features and high-level features extracted by the second encoder for the haze image; Both the training set and the verification set of the bidirectional GAN network model include pairs of clear images and fog images. 4 . The ultra-lightweight image dehazing and recognition network model according to claim 3 , wherein the first encoder and the second encoder respectively comprise first convolution blocks connected in sequence. 5 . , the second convolution block and the first coupled residual block; The first decoder and the second decoder respectively include a second coupled residual block, a first-sized convolutional block and a second-sized convolutional block connected in sequence; The output of the first coupled residual block is connected to the input of the shared latent space, and the output of the shared latent space is connected to the input of the second coupled residual block; The output terminal of the first convolution block is jump connected to the input terminal of the second step size convolution block; The output terminal of the second convolution block is jump connected to the input terminal of the first length convolution block. 5 . The ultra-lightweight image dehazing and recognition network model according to claim 4 , wherein the first coupled residual block and the second coupled residual block are respectively composed of a plurality of sub-residuals. 6 . block cascade composition; The sub-residual block at any stage includes a first convolution layer, an activation function layer and a second convolution layer that are connected in sequence, and are used to process the output results of the sub-residual block of the previous stage and the sub-residual of the previous stage. The output result of the block, and the processing result is output to the next-level sub-residual block and the next-level sub-residual block. 6. The ultra-lightweight image dehazing and identification network model according to claim 3, wherein the first discriminator and the second discriminator both comprise three layers of discriminant networks and one layer of granularity Discriminant network, any layer The discriminant network includes three convolutional layers and one activation function layer. 7. a kind of ultra-lightweight image dehazing and identification network model according to claim 3, is characterized in that, described two-way GAN network model is optimized by following loss function: generate confrontation loss function, MSE loss function and total. Variational loss function. 8. A picture dehazing and identification method, wherein the method comprises: Extract the depth information corresponding to different pixels in the image to be dehazed; performing depth processing on the picture to be dehazed according to the depth information; Input the deeply processed image to be dehazed into a pre-built ultra-lightweight image dehazing and recognition network model; Obtain the image dehazing and recognition results output by the ultra-lightweight image dehazing and recognition network model. 9. The image dehazing and identification method according to claim 8, wherein the extraction of depth information corresponding to different pixels in the picture to be dehazed comprises: Perform format conversion on the picture to be dehazed, and extract the brightness and saturation of the picture to be dehazed; According to the brightness and saturation, depth information corresponding to different pixels in the image to be dehazed is generated. 10. A computer equipment, characterized in that, comprising: memory for storing computer programs; The processor is configured to implement the steps of the image dehazing and identification method according to any one of claims 8 or 9 when executing the computer program.

Images