CN116645569B - A method and system for colorizing infrared images based on generative adversarial networks - Google Patents

Abstract

The present invention belongs to the field of image processing technology, and is particularly a method for colorizing infrared images based on a generative adversarial network. The specific steps are as follows: constructing a network model: constructing a generative adversarial network including a generator and a discriminator; preparing a data set: pre-training the generative adversarial network according to a first infrared image data set; training the network model: training the network model using the first infrared image data set until a preset threshold is reached; fine-tuning the model: retraining and fine-tuning the network model using a second infrared image data set to obtain a final model; saving the model: solidifying the parameters of the final model and saving the model. The network structure adopted by the present invention is a generative adversarial network structure, which utilizes the game characteristics between the generator and the discriminator to strengthen the extraction of deep information of the image, enhance the naturalness and authenticity of the colorized image, and dynamically improve the quality of the colorized image.

Description

A method and system for colorizing infrared images based on generative adversarial networks

Technical Field

The present invention relates to the technical field of image processing, and in particular to an infrared image colorization method and system based on a generative adversarial network.

Background Art

Infrared images are of great significance in many complex scenes. For example, in haze or rainy and foggy weather, infrared images have stronger penetrating power than visible light and can penetrate denser air to capture clear images. At the same time, in military operations, infrared images can also better identify camouflaged enemies, thereby increasing the hope of victory in the war. However, infrared images are not conducive to people's direct viewing and obtaining information from them, which will reduce people's recognition ability and increase the difficulty of people's understanding of images. The human eye's recognition ability for color scenes is much greater than that for grayscale scenes. Therefore, converting infrared images into colorized images is not only very practical, but also has very broad applications. It can provide multiple solutions for a variety of complex scenes. However, unlike the grayscale image colorization algorithm that only estimates the image chromaticity information, infrared image colorization requires simultaneous estimation of the image's brightness and chromaticity information. At the same time, the thermal characteristics of infrared images are not necessarily related to the characteristics of their corresponding visible light images, so the results of infrared image colorization have serious problems of blurred details and texture distortion. These objective problems have exacerbated the difficulty of realizing infrared image colorization.

The Chinese patent application number is "CN202210199669.2", and its name is "Multi-scale neural network infrared image colorization method based on attention mechanism". This method first uses a two-dimensional convolutional neural network to extract features of the input infrared image pair at different resolutions; the processed image is input into the backbone network; the input image is encoded using the backbone network to obtain the input features; the extracted high-dimensional feature information is then condensed through the attention mechanism; finally, the multi-scale information is fused to obtain a predicted color infrared image; the colorized image obtained by this method cannot accurately retain structural information and semantic details, and has poor image quality, low contrast, unnatural colors, and does not conform to the visual effects of the human eye. At the same time, the computational complexity is high and the efficiency is low; therefore, how to overcome the above-mentioned defects is a problem that technicians in this field urgently need to solve.

Summary of the invention

1. Technical issues to be solved

In view of the shortcomings of the prior art, the present invention provides an infrared image colorization method and system based on a generative adversarial network, which solves the problems of poor image quality, low contrast, unnatural colors and low efficiency obtained by the existing infrared image colorization methods.

(II) Technical solution

In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

A method for colorizing infrared images based on a generative adversarial network, the specific steps are as follows:

Build a network model: Build a generative adversarial network including a generator and a discriminator;

Prepare the dataset: pre-train the generative adversarial network based on the first infrared image dataset;

Training the network model: using the first infrared image data set to train the network model until a preset threshold is reached;

Fine-tuning the model: retraining and fine-tuning the network model using the second infrared image dataset to obtain the final model;

Save model: Solidify the parameters of the final model and save the model.

Furthermore, the generator includes a multi-layer dense module, a downsampling module, a feature fusion module, a color-aware attention module, an upsampling module, and an image reconstruction module;

The multi-layer dense module is used to extract features from images using convolution blocks;

The downsampling module is composed of multiple layers of dense modules and convolution blocks; it is used to continuously reduce the size of feature maps and extract deep semantic information of images;

The feature fusion module is used to reduce the information loss caused by encoder downsampling;

The color-aware attention module is used to effectively improve the colorization quality of the network by enhancing important features and suppressing unnecessary features. At the same time, it strengthens the perceptual information, captures more semantic details, and focuses on more key objects;

The upsampling module is used to gradually restore the size of the feature map;

The image reconstruction module is used to reconstruct the infrared colorized image using a convolution block and a T-type function.

Furthermore, the discriminator includes multiple convolutional blocks, a color-aware attention module, a normalization layer, and an L-type function to enhance the perceptual information and facilitate fast convergence.

Furthermore, the first infrared image dataset is a KAIST dataset.

Furthermore, the preset thresholds in the training network model include a preset value of the loss function and a preset value of the number of training times.

Furthermore, the loss function is a composite loss function. The loss functions adopted by the generator include synthetic loss, adversarial loss, feature loss, overall change loss and gradient loss; the discriminator adopts adversarial loss.

Furthermore, the process of training the network model also includes evaluating the quality of the algorithm colorization results and the degree of image distortion through evaluation indicators.

Further, the second infrared image data set is a FLIR data set.

An infrared image colorization system based on a generative adversarial network, the system comprising:

An image acquisition module, used to acquire training data and a deep neural network; the training data includes input image information and output image information;

An image processing module, used for preprocessing each image in the training data; the preprocessing includes geometric correction, image enhancement and image filtering;

A feature extraction module, used for extracting features from each of the images to obtain a feature map set for each of the images; the feature map set includes a plurality of feature maps of different sizes;

A model training module, used for performing supervised model training on a feature map of a specified size in a feature map set of each image to obtain an infrared image colorization model;

An image colorization module, used for inputting the infrared image to be colorized into the infrared image colorization model to perform infrared image colorization processing;

Storage medium, used for storing infrared image colorization system;

Furthermore, the image processing module is also used for data enhancement of the training data set, and commonly used methods include: image cropping, image flipping and image translation;

Furthermore, the feature extraction module is also used for extracting a feature map of a specified size in the feature map set of each image, wherein there are no less than 2 feature maps of the specified size of the model to be trained, and the specified size is 1/8 of the original image size;

Furthermore, the model training module is also used to resize each image from any size of the dataset to a fixed size of 256×256; both the generator and the discriminator are trained with a batch size of 1 for 200 epochs; initially, the learning rate is set to 0.0002 in the first 100 epochs, and the learning rate is linearly decreased to 0 in the next 100 epochs; the number of filters in the first convolutional layer in the generator and the discriminator is set to 64; we use the Adam optimizer with β ₁ = 0.5, β ₂ = 0.999; the discriminator and the generator are trained alternately until the model converges;

Furthermore, the image colorization module is also used to obtain the image to be colored, input the information of the image to be colored into the infrared image colorization model, and generate the infrared image colorization result by using the forward propagation algorithm.

(III) Beneficial effects

Compared with the prior art, the present invention provides an infrared image colorization method and system based on a generative adversarial network, which has the following beneficial effects:

The network structure adopted in the present invention is a generative adversarial network structure, which utilizes the game characteristics between the generator and the discriminator to strengthen the extraction of deep information of the image, enhance the naturalness and authenticity of the colorized image, and dynamically improve the quality of the colorized image.

The present invention adopts a dense residual structure in the multi-layer dense module 1, the down-sampling module 1, the down-sampling module 2, the down-sampling module 3, the up-sampling module 1, the up-sampling module 2, the up-sampling module 3 and the multi-layer dense module 2 to extract features from the feature map, extracts local and context information with fewer parameters and computational costs, enhances the feature extraction capability and improves the detail recovery capability, and produces a higher quality color infrared image.

The present invention adopts a color-aware attention module in upsampling module one, upsampling module two and upsampling module three to effectively improve the colorization quality of the network by enhancing important features and suppressing unnecessary features. At the same time, it strengthens perceptual information, captures more semantic details, and pays attention to more key objects.

The feature fusion module proposed in the present invention is applied between the encoder and the decoder, which reduces the information loss caused by the downsampling of the encoder and improves the prediction ability of the subtle colors of the image.

The present invention proposes a composite loss function composed of synthesis loss, adversarial loss, feature loss, overall variation loss and gradient loss, which can improve the quality of colorized images, generate fine local details, and restore semantic and texture information.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of an infrared image colorization method based on a generative adversarial network;

FIG2 is a structural diagram of a generator and a discriminator of a generative adversarial network in an infrared image colorization method based on a generative adversarial network;

FIG3 is a schematic diagram of the specific composition of each module in the downsampling module 1, the downsampling module 2 and the downsampling module 3 according to the present invention;

FIG4 is a schematic diagram of the specific composition of each module in the upsampling module 1, the upsampling module 2 and the upsampling module 3 according to the present invention;

FIG5 is a schematic diagram showing the specific composition of all multi-layer dense modules described in the present invention;

FIG6 is a schematic diagram of the specific composition of the feature fusion module of the present invention;

FIG7 is a schematic diagram of the specific composition of the color perception attention module of the present invention;

FIG8 is a schematic diagram of the specific structure of all convolution blocks described in the present invention;

FIG9 is a schematic diagram showing the specific structure of all transposed convolution blocks described in the present invention;

FIG10 is a schematic diagram showing a comparison of related indicators of the method proposed in the present invention;

FIG11 is a schematic diagram of the main modules of the infrared image colorization system of the present invention;

FIG. 12 is a schematic diagram of the internal structure of an electronic device for implementing the infrared image colorization method described in the present invention.

DETAILED DESCRIPTION

The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

Example

Example 1

As shown in FIG1 , a flowchart of an infrared image colorization method based on a generative adversarial network is shown. The method specifically includes the following steps:

Step 1, build a network model; the entire generative adversarial network includes a generator and a discriminator; the generator consists of a multi-layer dense module 1, a downsampling module 1, a downsampling module 2, a downsampling module 3, a feature fusion module, an upsampling module 1, an upsampling module 2, an upsampling module 3, a splicing operation, a multi-layer dense module 2, and an image reconstruction module; the discriminator consists of a convolutional layer 1, a convolutional layer 2, a convolutional layer 3, a convolutional layer 4, a convolutional layer 5, a color perception attention module, a normalization layer, and an L-type function. A color perception attention module is added to the middle convolutional layer to enhance the perception information, which is conducive to rapid convergence; the infrared image generated by the generator is The colorized image and the visible light color image in the data set are input into the discriminator, and the discriminator outputs the true and false probability information to determine whether the input image is real; the multi-layer dense module 1 performs shallow feature extraction on the image; the downsampling module 1, the downsampling module 2 and the downsampling module 3 continuously reduce the size of the feature map and extract the deep semantic information of the image; the feature fusion module reduces the information loss caused by the encoder downsampling; the upsampling module 1, the upsampling module 2 and the upsampling module 3 use the color perception attention module to make the network pay more attention to the semantic information and color information of the image, capture more semantic details, and gradually restore the size of the feature map; splicing The operation reuses the shallow information and the deep information for feature reuse; the multi-layer dense module 2 and the image reconstruction module reconstruct the infrared colorized image; the downsampling module consists of a multi-layer dense module and a convolution block; the upsampling module consists of a multi-layer dense module, a transposed convolution block, a convolution block and a color perception attention module; the multi-layer dense module consists of a convolution block 1, a convolution block 2, a convolution block 3, a convolution block 4, a convolution block 5, a convolution block 6, a convolution block 7, a convolution block 8 and a splicing operation; the feature fusion module consists of a convolution block 1, a convolution block 2, a convolution block 3, a convolution block 4, a splicing operation, an addition operation and a bilinear interpolation upsampling operation; the color perception attention The intention module consists of convolution block 1, convolution block 2, convolution block 3, convolution block 4, convolution block 5, convolution block 6, convolution block 7, convolution block 8, convolution block 9, addition operation, multiplication operation, Softmax operation, average pooling operation and expansion operation; each transposed convolution block consists of a transposed convolution layer, a normalization layer and an L-type function, and the size of the convolution kernel is unified to n×n; the image reconstruction module consists of convolution layer 1, convolution layer 2 and a T-type function; each convolution block consists of a convolution layer, a normalization layer and an L-type function, and the size of the convolution kernel is unified to n×n; finally, the size of the feature map is consistent with the size of the input image;

Step 2, prepare the data set; first train the entire generative adversarial network with the infrared image data set 1; during the pre-training process, the infrared image data set uses the KAIST data set; supervised training is performed on the images in the data set;

Step 3, training the network model; training the infrared image colorization model, preprocessing the data set prepared in step 2, adjusting the size of each image in the data set, fixing the size of the input image, and inputting the processed data set into the network model constructed in step 1 for training;

Step 4, select the minimized loss function value and the optimal evaluation index; by minimizing the loss function of the network output image and the label, until the number of training times reaches the set threshold or the value of the loss function reaches the set range, the model parameters can be considered to have been pre-trained and the model parameters can be saved; at the same time, the optimal evaluation index is selected to measure the accuracy of the algorithm and evaluate the performance of the system; during the training process, the loss function uses a composite loss function, the generator uses synthetic loss, adversarial loss, feature loss, overall change loss and gradient loss, and the discriminator uses adversarial loss; the choice of loss function affects the quality of the model, can truly reflect the difference between the predicted value and the true value, and can correctly feedback the quality of the model; suitable evaluation indicators include peak signal-to-noise ratio (PSNR), structural similarity (SSIM), perceptual image similarity (LPIPS) and natural image quality evaluation (NIQE), which can effectively evaluate the quality of the algorithm colorization results and the degree of image distortion, and measure the role of the colorization network;

Step 5, fine-tune the model; use infrared image dataset 2 to train and fine-tune the model to obtain stable and usable model parameters, further improve the model's infrared image colorization capability, and ultimately make the model more effective in infrared image colorization; use the FLIR dataset in the process of fine-tuning the model parameters;

Step 6, save the model; solidify the finalized model parameters;

If the infrared image is colorized, the final colorized image can be obtained by directly inputting the image into the network;

The present invention also provides an infrared image colorization system, the system comprising:

An image acquisition module, used to acquire training data and a deep neural network; the training data includes input image information and output image information;

An image processing module, used for preprocessing each image in the training data; the preprocessing includes geometric correction, image enhancement and image filtering;

A feature extraction module, used for extracting features from each of the images to obtain a feature map set for each of the images; the feature map set includes a plurality of feature maps of different sizes;

A model training module, used for performing supervised model training on a feature map of a specified size in a feature map set of each image to obtain an infrared image colorization model;

An image colorization module, used for inputting the infrared image to be colorized into the infrared image colorization model to perform infrared image colorization processing;

Storage medium, used for storing infrared image colorization system;

The present invention also provides an electronic device for colorizing infrared images, the device comprising: one or more processors; a storage system for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the infrared image colorization method provided by the present invention;

The present invention also provides a computer-readable storage medium, characterized in that it stores a computer program, and running the computer program can execute the infrared image colorization method provided by the present invention.

Example 2

As shown in FIG1 , a method for colorizing an infrared image based on a generative adversarial network specifically includes the following steps:

Step 1, build a network model;

As shown in Figure 2, the generator consists of a multi-layer dense module 1, a downsampling module 1, a downsampling module 2, a downsampling module 3, a feature fusion module, an upsampling module 1, an upsampling module 2, an upsampling module 3, a splicing operation, a multi-layer dense module 2 and an image reconstruction module; the downsampling module consists of a dense module and a convolution block, and the specific structure of each downsampling module is shown in Figure 3; the upsampling module consists of a multi-layer dense module, a transposed convolution block, a convolution block and a color perception attention module, and the specific structure of each upsampling module is shown in Figure 4; the multi-layer dense module consists of a convolution block 1, a convolution block 2, a convolution block 3, a convolution block 4, a convolution block 5, a convolution block 6, a convolution block 7, a convolution block 8 and a splicing operation, the convolution kernel size is 3×3, and the step size is 1. The specific structure of each multi-layer dense module is shown in Figure 5; the feature fusion module consists of a convolution block 1, a convolution block 2, a convolution block 3, a convolution block 4, a splicing operation, an addition operation and a bilinear interpolation upsampling operation, and the convolution kernel size is 3 ×3, the step sizes are 2 and 1 respectively. The specific structure of each feature fusion module is shown in Figure 6; the color perception attention module consists of convolution block 1, convolution block 2, convolution block 3, convolution block 4, convolution block 5, convolution block 6, convolution block 7, convolution block 8, convolution block 9, addition operation, multiplication operation, Softmax operation, average pooling operation and expansion operation. The convolution kernel sizes are 1×1 and 3×3 respectively, the step size is 1, the expansion rate of convolution block 3 and convolution block 4 is 3, and each color perception The specific structure of the attention module is shown in Figure 7; the transposed convolution block consists of a transposed convolution layer, a normalization layer and an L-type function, the convolution kernel size is 4×4, and the step length is 2. The specific structure of each transposed convolution block is shown in Figure 9; the image reconstruction module consists of a convolution layer 1, a convolution layer 2 and a T-type function, the convolution kernel size is 1×1, and the step length is 1; the convolution block consists of a convolution layer, a normalization layer and an L-type function, the convolution kernel size and step length depend on the situation, and the specific structure of each convolution block is shown in Figure 8;

The discriminator consists of convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4, convolutional layer 5, color perception attention module, normalization layer and L-type function. The color perception attention module is added to the middle convolutional layer to strengthen the perception information, which is conducive to rapid convergence. The infrared colorized image generated by the generator and the visible light image in the data set are input into the discriminator, and the discriminator outputs true and false probability information to determine whether the input image is real.

In general, the colorization process is to input the infrared image, extract the features through 3 downsampling operations, and then restore the feature map size through 3 upsampling operations to reconstruct the infrared colorized image. Finally, the output colorized image and the visible light image are input into the discriminator to determine whether it is real.

In order to ensure the robustness of the network, retain more structural information, and fully extract image features, the present invention uses three activation functions, namely L-type function, T-type function and S-type function. The last layer of the generator is the T-type function, and the last layer of the discriminator is the S-type function. In addition, all loss functions of the generator and the discriminator are L-type functions; the definitions of L-type function, T-type function and S-type function are as follows:

Step 2, prepare the data set; the infrared image data set uses the KAIST data set; the KAIST pedestrian data set includes a total of 95,328 images, each of which contains two versions: RGB color image and infrared image; the data set captures various regular traffic scenes including campus, streets and countryside during the day and night; the image size is 640×480; although the KAIST data set itself indicates that it is registered, after careful observation, we found that some infrared images and visible light images still have some deviations; and because the KAIST data set is taken from continuous frame images of the video, the adjacent images are not much different; after cleaning operations, we selected 4755 daytime images and 2846 nighttime images in the training set; 1455 daytime images and 797 nighttime images in the test set; we only use the daytime training set for training; resize these 4755 images to 256×256 as the input of the entire network; adversarial training of the KAIST data set can determine a set of initialization parameters to speed up the subsequent network training process;

In step 3, the image of the data set is enhanced, a random diffraction transformation is performed on the same image, and it is cropped to the size of the input image as the input of the entire network, and the annotated image in the data set is used as a label; wherein the random size and position can be achieved through a software algorithm; wherein the annotated image in the data set is used as a label in order to allow the network to learn better feature extraction capabilities, and ultimately achieve a better colorization effect;

The output of the network in step 4 and the label calculate the loss function, and achieve a better fusion effect by minimizing the loss function; during the training process, the loss function uses a composite loss function, the generator uses synthetic loss, adversarial loss, feature loss, overall change loss and gradient loss, and the discriminator uses adversarial loss;

The synthesis loss is actually the L1 loss. By adding the synthesis loss, the brightness and contrast differences between the color image and the ground truth can be effectively minimized. If GAN pays too much attention to the synthesis loss, the brightness and contrast in the infrared image will be lost. In order to prevent the generator from over-representing the pixel-to-pixel relationship, an appropriate weight of the synthesis loss will be added. The synthesis loss can be expressed as:

Where W and H represent the height and width of the infrared image, respectively, I _ir represents the input infrared image, I _vis represents the Ground Truth, G(·) represents the generator, and ||·|| ₁ represents the L1 norm;

The colorization result using synthetic loss will lose some details; in order to encourage the network to output color results with more realistic details, adversarial loss is used; adversarial loss is used to make the colorized image indistinguishable from the ground truth, defined as:

The input infrared image I _ir is not only the input of the generator, but also the input of the discriminator as a conditional item;

L _synthesized and L _adv sometimes fail to ensure consistency between perceptual quality and objective metrics; to alleviate this problem, we use feature loss, which compares the differences between feature maps extracted by tools such as dedicated pre-trained models; in this paper, a pre-trained VGG-16CNN model is used to identify and extract features of the input image; to synthesize feature loss, the VGG-16CNN model extracts feature maps of the color result and GroundTruth, and then calculates the Euclidean distance between the pair of feature maps; the feature loss is expressed as:

Where φ _n (·) represents the feature map of the nth layer in the VGG-16 network, and C _n , H _n and W _n represent the number of channels, height and width of the layer respectively;

In addition to using feature loss to recover high-level content, we also adopt overall variation loss to enhance the spatial smoothness of color infrared images. The overall variation loss is defined as:

where |·| represents the element-wise absolute value of the given input;

We use gradient loss to enhance the quality of colorized images. The network uses the gradient loss of visible light images and the gradient loss of infrared images to constrain the network and obtain the texture information and brightness information of the image. Gradient loss is conducive to faster convergence of the network. The gradient loss of visible light images and the gradient loss of infrared images are defined as:

L _G2 =L _gradient1 +L _gradient2

in represents the gradient;

Therefore the total loss of the generator is defined as:

L _total ＝λ _adv L _adv +λ _feature L _feature +λ _synthesized L _synthesized +λ _tv L _tv +λ _G2 L _G2

Where λ _adv , λ _feature , λ _synthesized , λ _tv and λ _G2 represent the weights controlling different loss shares in the complete loss function; the weights are set based on preliminary experiments on the training dataset;

The loss function of the discriminator is defined as:

By optimizing the generator and discriminator loss functions, the network can learn clearer edges and more detailed textures, making the colorized images more natural, more realistic, and more visually appealing.

The appropriate evaluation indicators in step 4 are peak signal-to-noise ratio (PSNR), structural similarity (SSIM), perceptual image similarity (LPIPS) and natural image quality evaluation (NIQE). The peak signal-to-noise ratio is based on the error between corresponding pixels, that is, an error-sensitive image quality evaluation; structural similarity measures image similarity from three aspects: brightness, contrast and structure, and is an indicator used to measure the similarity between two digital images; perceptual image similarity learns the reverse mapping of the generated image to the ground truth to force the generator to learn the reverse mapping of reconstructing the real image from the fake image, and prioritizes the perceptual similarity between them; natural image quality evaluation is based on a set of "quality-aware" features and fits them to a multivariate Gaussian model; the quality-aware features are derived from a simple but highly regularized natural statistical feature model; then, the natural image quality evaluation indicator of a given test image is expressed as the distance between the multivariate Gaussian model of the natural statistical features extracted from the test image and the multivariate Gaussian model of the quality-aware features extracted from the natural image corpus; peak signal-to-noise ratio, structural similarity, perceptual image similarity and natural image quality evaluation are defined as follows:

Where μ _x and μ _y represent the mean and variance of images x and y respectively. and Represent the standard deviation of images x and y respectively, σ _xy represents the covariance of images x and y, C ₁ and C ₂ are constants, d is the distance between x ₀ and x, w _l is a trainable weight parameter, v ₁ ,v ₂ ,∑ ₁ and ∑ ₂ represent the mean vector and covariance matrix of the natural multivariate Gaussian model and the distorted image multivariate Gaussian model respectively;

The number of training times is set to 200, the learning rate of the first 100 training processes is set to 0.0002, and the learning rate of the next 100 training processes is gradually reduced from 0.0002 to 0; the upper limit of the number of images input to the network each time is mainly determined by the performance of the computer graphics processor. Generally, the number of images input to the network each time is in the range of 8-16, which can make the network training more stable and the training results better, and can ensure the rapid fitting of the network; the Adam optimizer is selected as the network parameter optimizer; its advantages are mainly simple implementation, efficient calculation, low memory requirements, and parameter updates are not affected by the scaling transformation of the gradient, making the parameters relatively stable; when the discriminator's ability to judge fake images is balanced with the generator's ability to generate images that deceive the discriminator, the network is considered to have been basically trained;

Step 5, fine-tune the model; use the infrared image dataset 2 to train and fine-tune the model, and use the FLIR dataset in the process of fine-tuning the model parameters; the FLIR dataset has 8862 unaligned visible light and infrared image pairs, containing a variety of scenes, such as roads, vehicles, pedestrians, etc.; these images are very representative scenes in FLIR videos; in this dataset, we select feature points through the cpselect algorithm to form a feature point matrix, and manually align the FLIR dataset; after screening and alignment, we use 3918 image pairs for training and 428 image pairs for testing;

After the network training is completed in step 6, all parameters in the network need to be saved, and then the infrared image to be colored is input into the network to obtain a colored image; the network has no requirements on the size of the input image, and any size is acceptable;

Among them, the implementation of convolution, activation function, splicing operation and batch normalization are algorithms well known to those skilled in the art, and the specific processes and methods can be found in corresponding textbooks or technical literature;

The present invention constructs an infrared image colorization method based on a generative adversarial network, which can directly generate a colorized image from an infrared image without going through other intermediate steps, thus avoiding the manual design of relevant colorization rules; under the same conditions, the relevant indexes of the image obtained by calculating the existing method are further verified to verify the feasibility and superiority of the method; the comparison of the relevant indexes of the existing technology and the method proposed in the present invention is shown in FIG10;

As can be seen from FIG10 , the method proposed in the present invention has a higher peak signal-to-noise ratio, higher structural similarity, lower perceived image similarity, lower natural image quality assessment, and fewer generator parameters than the existing method. These indicators further illustrate that the method proposed in the present invention has better colorization quality and lower computational complexity.

As shown in FIG11 , the present invention also provides an infrared image colorization system, which mainly includes an image acquisition module, an image processing module, a feature extraction module, a model training module, an image colorization module and a storage medium;

An image acquisition module, used to acquire training data and a deep neural network; the training data includes input image information and output image information;

An image processing module, used for preprocessing each image in the training data; the preprocessing includes geometric correction, image enhancement and image filtering;

A feature extraction module, used for extracting features from each of the images to obtain a feature map set for each of the images; the feature map set includes a plurality of feature maps of different sizes;

A model training module, used for performing supervised model training on a feature map of a specified size in a feature map set of each image to obtain an infrared image colorization model;

An image colorization module, used for inputting the image to be colorized into the infrared image colorization model to perform infrared image colorization processing;

Storage medium, used for storing infrared image colorization system;

Furthermore, the image processing module is also used for data enhancement of the training data set, and commonly used methods include: image cropping, image flipping and image translation;

Furthermore, the feature extraction module is also used for extracting a feature map of a specified size in the feature map set of each image, wherein there are no less than 2 feature maps of the specified size of the model to be trained, and the specified size is 1/8 of the original image size;

Furthermore, the model training module is also used to resize each image from any size of the dataset to a fixed size of 256×256; both the generator and the discriminator are trained with a batch size of 1 for 200 epochs; initially, the learning rate is set to 0.0002 in the first 100 epochs, and the learning rate is linearly decreased to 0 in the next 100 epochs; the number of filters in the first convolutional layer in the generator and the discriminator is set to 64; we use the Adam optimizer with β ₁ = 0.5, β ₂ = 0.999; the discriminator and the generator are trained alternately until the model converges;

Furthermore, the image colorization module is also used to obtain the image to be colored, input the information of the image to be colored into the infrared image colorization model, and generate the infrared image colorization result by using the forward propagation algorithm;

As shown in FIG12 , the present invention also provides an infrared image colorization electronic device, which mainly includes a memory, a processor, a communication interface and a bus; wherein the memory, the processor and the communication interface are connected to each other through the bus;

The memory may be a ROM, a static storage device, a dynamic storage device or a RAM; the memory may store a program, and when the program stored in the memory is executed by the processor, the processor and the communication interface are used to execute the various steps of the training method for the infrared image colorization network of the embodiment of the present invention;

The processor may be a CPU, a microprocessor, an ASIC, a GPU or one or more integrated circuits, and is used to execute relevant programs to implement the functions required to be performed by the units in the infrared image colorization training system of the present invention, or to perform the infrared image colorization training method of the present invention;

The processor may also be an integrated circuit chip with signal processing capability; in the implementation process, each step of the infrared image colorization training method of the present invention may be completed by hardware integrated logic circuits or software instructions in the processor; the above-mentioned processor may also be a general-purpose processor, DSP, ASIC, FPGA or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components; the infrared image colorization method, steps and logic block diagram of the present invention may be implemented or executed; the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.; the steps of the infrared image colorization method of the present invention may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in the decoding processor; the software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc.; the storage medium is located in a memory, the processor reads the information in the memory, and in combination with its hardware completes the functions required to be executed by the units included in the infrared image colorization training system of the present invention, or executes the infrared image colorization training method of the present invention;

The communication interface uses a transceiver system such as, but not limited to, a transceiver to achieve communication between the system and other devices or communication networks; for example, the image to be processed or the initial feature map of the image to be processed can be obtained through the communication interface;

A bus may include a pathway that transfers information between various components of a system (e.g., memory, processor, communication interface);

The present invention also provides a computer-readable storage medium for colorization of infrared images, which may be a computer-readable storage medium included in the system in the above-mentioned embodiment; or a computer-readable storage medium that exists independently and is not assembled into a device; the computer-readable storage medium stores one or more programs, and the programs are used by one or more processors to execute the method described in the present invention;

It should be noted that although the electronic device shown in FIG12 only shows a memory, a processor, and a communication interface, during the specific implementation process, those skilled in the art should understand that the system also includes other devices necessary for normal operation; at the same time, according to specific needs, those skilled in the art should understand that the system may also include hardware devices for implementing other additional functions; in addition, those skilled in the art should understand that the system may also include only the devices necessary to implement the embodiments of the present invention, without necessarily including all the devices shown in FIG12.

Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims (7)

1. A method for colorizing infrared images based on a generative adversarial network, characterized in that the specific steps are: Build a network model: Build a generative adversarial network including a generator and a discriminator; The generator includes a multi-layer dense module, a downsampling module, a feature fusion module, a color-aware attention module, an upsampling module, and an image reconstruction module; The multi-layer dense module is used to extract features from images using convolution blocks; The multi-layer dense module consists of convolution block 1, convolution block 2, convolution block 3, convolution block 4, convolution block 5, convolution block 6, convolution block 7, convolution block 8 and splicing operation. The convolution kernel size is 3× 3 and the step size is 1. The downsampling module is composed of multiple layers of dense modules and convolution blocks; it is used to continuously reduce the size of feature maps and extract deep semantic information of images; The feature fusion module is used to reduce the information loss caused by encoder downsampling; The color-aware attention module is used to effectively improve the colorization quality of the network by enhancing important features and suppressing unnecessary features. At the same time, it strengthens the perceptual information, captures more semantic details, and focuses on more key objects; The color perception attention module consists of convolutional blocks 1, 2, 3, 4, 5, 6, 7, 8, 9, addition, multiplication, Softmax, average pooling and expansion. The convolution kernel sizes are 1 × 1 and 3 × 3 respectively, with a step size of 1. The expansion rates of convolutional blocks 3 and 4 are 3. The upsampling module is used to gradually restore the size of the feature map; The convolutional block consists of a convolutional layer, a normalization layer, and an L-type function; The image reconstruction module is used to reconstruct the infrared colorized image using a convolution block and a T-type function; The discriminator includes multiple convolutional blocks, color-aware attention modules, normalization layers, and L-type functions to enhance perceptual information and facilitate fast convergence. Prepare the dataset: pre-train the generative adversarial network based on the first infrared image dataset; Training the network model: using the first infrared image data set to train the network model until a preset threshold is reached; Fine-tuning the model: retraining and fine-tuning the network model using the second infrared image dataset to obtain the final model; Save model: Solidify the parameters of the final model and save the model. 2. According to claim 1, a method for infrared image colorization based on a generative adversarial network is characterized in that the first infrared image dataset is a KAIST dataset. 3. According to claim 1, a method for colorizing infrared images based on a generative adversarial network is characterized in that the preset thresholds in the training network model include a preset value of the loss function and a preset value of the number of training times. 4. According to claim 1, a method for colorizing infrared images based on a generative adversarial network is characterized in that the loss function is a composite loss function, the loss function used by the generator includes synthetic loss, adversarial loss, feature loss, overall change loss and gradient loss; the discriminator uses adversarial loss. 5. According to claim 1, a method for colorizing infrared images based on a generative adversarial network is characterized in that the process of training the network model also includes evaluating the quality of the algorithm colorization results and the degree of image distortion through evaluation indicators. 6. According to claim 1, the infrared image colorization method based on generative adversarial network is characterized in that the second infrared image dataset is a FLIR dataset. 7. According to claim 1, an application system of an infrared image colorization method based on a generative adversarial network, characterized in that the system comprises: An image acquisition module, used to acquire training data and a deep neural network; the training data includes input image information and output image information; An image processing module, used for preprocessing each image in the training data; the preprocessing includes geometric correction, image enhancement and image filtering; A feature extraction module, used for extracting features from each of the images to obtain a feature map set for each of the images; the feature map set includes a plurality of feature maps of different sizes; A model training module, used for performing supervised model training on a feature map of a specified size in a feature map set of each image to obtain an infrared image colorization model; An image colorization module, used for inputting the infrared image to be colorized into the infrared image colorization model to perform infrared image colorization processing; Storage medium, used for storing infrared image colorization system; The image processing module is also used for data enhancement of the training data set, and common methods include: image cropping, image flipping and image translation; The feature extraction module is also used for the feature maps of the specified size in the feature map set of each image, and there are no less than 2 feature maps of the specified size for the model to be trained, and the specified size is 1/8 of the original image size; The model training module is also used to resize each image from an arbitrary size in the dataset to a fixed size of 256×256; both the generator and the discriminator are trained with a batch size of 1 for 200 epochs; initially, the learning rate is set to 0.0002 in the first 100 epochs, and the learning rate is linearly decreased to 0 in the next 100 epochs; the number of filters in the first convolutional layer in the generator and the discriminator is set to 64; we use the Adam optimizer, , ;The discriminator and generator are trained alternately until the model converges; The image colorization module is also used to obtain the image to be colored, input the information of the image to be colored into the infrared image colorization model, and generate the infrared image colorization result by using the forward propagation algorithm.