CN118505540A - Low-illumination image enhancement method based on random region hiding reconstruction - Google Patents

Abstract

The present invention discloses a low-light image enhancement method based on random region hidden reconstruction. The network is divided into two stages. The first stage of the network is based on a well-illuminated image data set. By training the network's ability to reconstruct noisy images after random regions are hidden, a denoising and reconstruction feature encoder for low-light noisy images is obtained; the second stage of the network, on the basis of inheriting the feature encoder model parameters obtained in the first stage, inputs the extracted different features into a multi-scale feature fusion module guided by the reconstruction features, so that the network can retain more image detail features while denoising. The present invention utilizes a more abundant source of image data sets under good lighting conditions, saving time and manpower for network training in shooting a large number of real paired low-light images; and by fusing the features extracted by different feature encoders, the network can quickly adapt to real low-light images under complex lighting, and obtain good low-light enhancement and denoising effects.

Description

A low-light image enhancement method based on random region hidden reconstruction

Technical Field

The invention belongs to the field of low-illumination image enhancement, and in particular relates to a low-illumination image enhancement method based on random region hidden reconstruction.

Background Art

With the advancement of science and technology, images are being used more and more widely in social production and life. Whether in the fields of medicine, education, entertainment, scientific research, industry, agriculture, etc., images play an important role. For example, medical images help doctors make diagnoses, and satellite images are used for climate forecasting and geographical research. High-quality images can not only provide a better visual experience, but also convey information more accurately and support decision-making more effectively. In practical applications, the environment, equipment and other reasons often cause low-quality images such as dark, noisy, and color deviation. Therefore, image enhancement technology is needed for post-processing to improve the quality and usability of images.

The main purpose of low-light enhancement is to improve the brightness and clarity of images in low-light environments. In traditional image processing methods, spatial domain algorithms such as histogram equalization and gamma correction directly process pixels to improve image brightness and contrast, but the processing results often amplify noise and bring problems such as artifacts and color distortion. Frequency domain algorithms such as low-pass filtering make images smoother, high-pass filtering improves the details at the edge junctions in the image, and homomorphic filtering can improve the details in the dark areas, but the overall processing effect on low-light images is not good, and parameters often need to be adjusted according to different lighting conditions, and the application effect is not good under complex lighting conditions in cities. Retinex theory is also widely used in low-light enhancement. Based on the trichromatic theory and color constancy, it believes that the reflection ability of an object to long, medium and short waves determines the color of the object, and is not affected by the intensity of the reflected light. Therefore, the original low-light image is decoupled into illumination component and reflection component, and the enhanced image is obtained by separating the reflection component from the original image. However, it does not work well for extremely dark environments and complex lighting environments, and often requires parameter adjustment, and the generalization is not strong.

With the development of neural networks and the improvement of computer computing power, image processing methods based on neural networks have been widely used in the field of vision. However, convolutional neural networks are limited by the size of the convolution kernel and have a small receptive field, so they are not good at capturing the global long-range relationships of images. In low-light enhancement tasks, it is easy to have uneven lighting problems in the enhancement results, and it is difficult to use useful information from non-adjacent blocks. The Transformer network has a good ability to capture global dependencies in global sequence data. ViT is the first network to apply the Transformer architecture to computer vision tasks, and has achieved the same performance as convolution or even better than convolution. However, as a sequence-to-sequence model, the Transformer uses pixels as sequence input, which will cause the input sequence to be too long, much longer than the sequence in natural language processing (e.g., 224 224=50176), which is very computationally intensive. In other words, existing low-light enhancement methods require a large number of real paired low-light image pairs for supervised network training, and shooting a large number of image pairs in low-light environments and good lighting conditions in the same scene is very labor-intensive and time-consuming. Currently, the number of such data sets is very small, which is not conducive to neural network training; at the same time, the Transformer network is a sequence-to-sequence model. When facing high-resolution images, it takes pixels as sequence input, which is very computationally intensive and requires high computing power; at the same time, existing low-light enhancement methods have the problem of limited low-light noise reduction capabilities, resulting in unclear edge details of the enhanced image. Therefore, it is urgent to propose a method that can quickly and effectively enhance low-light images.

Summary of the invention

In order to overcome the shortcomings of the prior art, the present invention provides a low-light image enhancement method based on random region hidden reconstruction. First, a richer set of images under good lighting conditions is used to perform low-light and random masking and noise processing on the input images under good lighting conditions. Then, the lost pixels hidden by the mask are reconstructed through a self-supervised learning task. While greatly reducing the amount of network calculations and saving computing power and time, the network's learning ability for the global information of low-light noisy images is improved, and the edge details can be better retained while the low-light noisy images are denoised.

In the second stage, the features extracted by different feature encoders can be better fused by using a multi-scale feature fusion module guided by reconstruction features. Compared with the limitations of the U-net convolutional denoising network, which is limited to a regular rectangular sliding window and the limited receptive field of the convolutional neural network, the feature fusion module is more flexible and has a wider range of sampling points to interact with, thereby extracting feature information that is more suitable for low-light noisy images. At the same time, the network parameters are fine-tuned using relatively few real low-light images, so that it can adapt to various real and complex low-light scenes more quickly, and can better achieve low-light enhancement and noise reduction than existing methods to obtain clear images. The present invention performs random mask hiding processing on the original image and then realizes the self-supervised learning task of image reconstruction, as well as the fusion of feature information of different features in the multi-scale feature fusion module guided by the reconstruction features, thereby reducing the amount of data and saving computer computing power, while improving the network's flexible processing ability for information of low-light noisy images.

In order to achieve the above object, the present invention adopts the following technical scheme:

The present invention discloses a low-light image enhancement method based on random region hidden reconstruction. The network used is divided into two stages for training. The two stages share the same feature extraction module (including a noise reduction feature encoder and a reconstruction feature encoder). The purpose of the first stage training is to train on a data set with richer sources of good photographic conditions to obtain a better feature extractor for low-light noisy images. The second stage inherits the feature extractor obtained by the first stage task, and then fuses the image features through a multi-scale feature fusion module guided by the reconstruction features. The network weights are adjusted through a small number of real low-light noisy images to obtain a low-light enhancement network with stronger generalization. Specifically, the following steps are included:

The specific steps of the first phase are:

Step (1): First, the image dataset under good lighting conditions is preprocessed by randomly adding noise and reducing brightness to obtain a synthetic low-light noisy image.

Step (2): After dividing the low-light noisy image obtained in step (1) into multiple image blocks of size 16×16, a random sequence is generated based on uniform distribution, the random values are sorted and mapped to the original image blocks, and the image blocks are masked at a ratio of 75%.

Step (3): Input the two images before and after the mask into a convolutional neural network with a convolution kernel size of 16×16 and a fully connected layer respectively to obtain the encoding result of an image of size 196×768. Use sine-cosine position coding to positionally encode each image block, and add the two to obtain the initial encoding information of each image block.

Step (4): Input the encoded information of the image into the denoising feature encoder and the reconstruction feature encoder to obtain a high-dimensional feature map;

Step (5): input the feature maps output from the feature encoders of the two branches into the multi-layer noise decoder and the multi-layer restoration decoder respectively;

Step (6): Output the decoded noise distribution map and the reconstructed map and compare them with the input image to calculate the reconstruction loss and perceptual loss.

The specific steps of the second phase are:

Step (7): Inherit the denoising feature encoder and reconstruction feature encoder obtained in the first stage task.

Step (8): Input the real scene low-light noisy image dataset into the two-branch feature encoder obtained in the first stage task for feature extraction.

Step (9): The feature maps finally obtained from the two branches are input together into the multi-scale feature fusion module guided by the reconstruction feature for feature fusion.

Step (10): Input the feature map obtained in step (9) into the low-light noise reduction decoder to obtain the result map after low-light noise reduction enhancement processing, and calculate the loss function with the true value to fine-tune the network weight parameters so that it can quickly adapt to various complex real noise images.

The beneficial effects of the present invention compared with the prior art are:

1. The first stage task uses a richer dataset under good lighting conditions to train the network, and obtains a feature extractor that can process low-light noisy images. Compared with the problem of noise amplification after low-light enhancement in existing enhancement methods, the present invention reduces the noise after enhancement while avoiding the problem of overfitting of the network due to the small number of real low-light image datasets.

2. Apply a 75% mask rate to the original image, so that the amount of data input to the network for feature extraction is only the original

The amount of calculation is greatly reduced, which saves computer computing power and speeds up the training of the network.

3. Use real low-light images with relatively small amounts of data to fine-tune the network weight parameters, so that the network can further adapt to real complex low-light scenes with better robustness and generalization.

4. Compared with the regular rectangular sliding window in the convolutional neural network, the present invention generates the offset of the sampling point relative to the feature point by reconstructing the features in the feature fusion model, thereby obtaining a more flexible interaction range.

5. Compared with the U-net denoising network that loses texture detail information due to multi-layer downsampling, the present invention uses sampling points of various scales when performing feature fusion, retains the details in the noisy image, is more suitable for noisy images, and reduces the image blur problem caused by denoising.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of the two-stage tasks in the present invention;

FIG2 is a schematic diagram of a feature fusion module guided by reconstruction features in the second stage task of the present invention;

FIG3 is a schematic diagram of the structure of a fusion module in a feature fusion module guided by reconstruction features in the second stage task of the present invention.

DETAILED DESCRIPTION

In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

The present invention first trains the neural network's ability to reconstruct images after random areas are masked and hidden, so that the neural network can learn detailed deep semantic information about low-light noisy images, and then uses real low-light image pairs to fine-tune the feature extraction network inherited from the first stage, and realizes the fusion of different types of features through multi-scale feature fusion guided by reconstruction features, thereby achieving more generalized low-light enhanced denoising that is adaptable to various real low-light conditions. It is suitable for scenarios with low visibility and provides clear images for systems such as autonomous driving, security monitoring, and target capture.

As shown in FIG1 , a low-light image enhancement method based on random region hidden reconstruction of the present invention adopts a network that uses two stages to obtain a low-light enhancement model, and the two stages share a denoising feature encoder and a reconstruction feature encoder.

The first stage includes the following steps:

Step (1): Gaussian noise and Poisson noise are applied to the image under good lighting conditions, and the image brightness is randomly reduced to generate a low-light noisy image with 3 channels, height 224 and width 224.

Step (2): Divide the 224×224 image into 14 14 image blocks, each image block size is (16,16). After dividing the low-light noisy image into image blocks, 75% of the image blocks are randomly selected for masking to obtain the masked low-light noisy image.

Step (3): Input the two images before and after the mask into a convolutional neural network with a convolution kernel size of 16×16 and a fully connected layer respectively to obtain the encoding result of an image of size 196×768. Use sine-cosine position coding to positionally encode each image block, and add the two to obtain the initial encoding information of each image block.

Step (4): Input the encoded information of the image into the denoising feature encoder and the reconstruction feature encoder to obtain a high-dimensional feature map. The feature encoder uses the encoder structure of the classic Transformer. The network structure of each feature encoder is composed of the following modules and processing layers in sequence: ① normalization layer; ② multi-head self-attention layer; ③ residual layer; ④ normalization layer; ⑤ multi-layer perceptron layer;

Step (5): Input the feature maps output from the feature encoders of the two branches into the multi-layer noise decoder and the multi-layer restoration decoder respectively. The multi-layer noise decoder and the multi-layer restoration decoder are both composed of 6 layers of feature decoders. Each layer of feature decoder is composed of the following modules and processing layers in sequence: multi-head self-attention layer, residual layer and normalization layer.

Step (6): Output the decoded noise distribution map and the reconstructed map and compare them with the input image to calculate the reconstruction loss and perceptual loss.

In the second stage, the feature decoder weights obtained in the first stage are used to initialize the corresponding modules of the second stage network, and the low-light enhancement network is trained using the following steps:

Step (7): Input the real low-light image into the second-stage network, repeat step (2) for encoding, and then inherit the denoising feature encoder and reconstruction feature encoder pre-trained in the first stage for feature extraction. The features finally extracted by the upper and lower branches are input into the multi-scale feature fusion module guided by the reconstruction features.

Step (8): As shown in FIG2, the features of the two branches are input into the multi-scale feature fusion module guided by the reconstruction feature. In the module, the fusion module is first passed. As shown in FIG3, in the fusion module, the denoised image features and the reconstructed image features are respectively input into different convolutional neural networks with convolution kernels of 3×3 and step sizes of 1, 2, and 2 for downsampling to obtain denoised image features and reconstructed image features of three scales. Then, the obtained multi-scale reconstructed image features are respectively input into convolutional neural networks with convolution kernels of 3×3 and step sizes of 1 to generate the offset of 9 irregular sampling points of each feature point in the corresponding denoised image feature map of each scale relative to the feature point. Then, according to the multi-scale offset map obtained under the guidance of the reconstruction feature, each feature point in the denoised image feature map of each scale is calculated with its 27 irregular sampling points (3 scales, 9 each) to generate an updated multi-scale denoised image feature map, which is then input into the next layer of feature fusion. After the last layer of fusion is completed, the small-scale denoised image feature map is spliced with the feature map by upsampling to obtain an updated denoised image feature map of the original size.

Step (9): Multiply the output of the fusion module in step (8) by the parameter matrices Wq and Wk to obtain feature maps Q and K, multiply the input denoised image features by the parameter matrix Wv to obtain the feature map V, perform cross-attention calculation on Q, K, and V, and output the obtained feature map as the new denoised image feature.

Step (10): After repeating steps (8) and (9) for N layers, the output denoised image features of the last layer are input into a multi-layer decoder consisting of 6 layers of low-light denoising decoders. Each layer of the decoder is composed of a multi-head self-attention layer, a residual layer, and a normalization layer. The final output is an enhanced result image, which is compared with the true value, and the L2 loss and perceptual loss are calculated. The overall network is fine-tuned so that the network can adapt well to the low-light real scene and obtain the enhanced result.

Preferably, the reconstruction loss and perceptual loss functions in step (6) and step (10) are L1 and L2 norms; during the first stage training, the Adam optimizer is used, wherein the exponential decay rate of the first-order moment estimate β ₁ =0.9, and the exponential decay rate of the second-order moment estimate β ₂ =0.999; the low-light denoising reconstruction model of the first stage is trained for 100 rounds, with an initial learning rate of 0.0006, which linearly decays to 0.0003 after 50 rounds; after the first stage training, the low-light enhancement denoising model of the second stage is fine-tuned for 30 rounds using a real low-light dataset, with a learning rate of 0.0002; to increase the availability of data, 500 pairs of real low-light datasets are randomly cropped and horizontally flipped during the second stage training to perform data enhancement to improve the low-light enhancement denoising performance of the second stage task.

It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims (6)

1. A low-light image enhancement method based on random region hidden reconstruction, characterized in that it comprises the following steps: Step 1: Get a data set of 10,000 images under good lighting conditions, the data type is color visible light images, and divide it into a training set and a test set; get a data set of 500 pairs of low-light noisy images and good-light images under the same real scene, the data type is color visible light images, and divide it into a training set and a test set; Step 2: First, a dataset of 10,000 images under good lighting conditions is used in the first-stage network. The dataset is preprocessed by adding noise and reducing brightness, and then the image is subjected to random region hiding processing before being input into the first-stage network. The network is then trained to restore and reconstruct the low-light noisy image after random hiding processing, and a denoising feature encoder and a reconstruction feature encoder with good information extraction capabilities for low-light noisy images are obtained. Step 3. Use 500 pairs of low-light noisy images of real scenes for the second-stage network, input the feature encoder obtained in step 2 for feature extraction, then input the obtained denoised image features and reconstructed image features into the reconstruction feature-guided multi-scale feature fusion for feature fusion, and then input the fused features into the low-light denoising decoder. Finally, use the paired good-light images in the same scene as the reference truth value for network training, so that the network parameters can quickly adapt to the real low-light scene, and obtain a low-light enhanced image with improved brightness and better quality after the network enhancement processing. 2. According to claim 1, a low-light image enhancement method based on random region hidden reconstruction is characterized in that the preprocessing in step 2 includes applying Gaussian noise and Poisson noise and uniformly reducing the image brightness; the random masking operation is to divide the low-light noisy image into multiple image blocks of size 16×16, generate a random sequence based on uniform distribution, sort the random values and map them to the original image blocks, and mask the image blocks at a ratio of 75%. 3. According to the low-illumination image enhancement method based on random region hidden reconstruction in claim 1, it is characterized in that the feature encoder of the branch of image restoration and reconstruction in step 2 only processes visible pixels that are not covered by the mask, and the decoder is responsible for performing the image reconstruction subtask by using the image features extracted by the reconstruction feature encoder and the covered pixel information. 4. According to the low-light image enhancement method based on random region hidden reconstruction according to claim 1, it is characterized in that the feature encoder in step 2 uses the encoder structure in the classic Transformer; finally, the feature maps of the two branches are respectively input into the feature decoder for noise localization and image restoration and reconstruction tasks, and the decoder uses the decoder structure in the Transformer. 5. According to the low-illumination image enhancement method based on random region hidden reconstruction described in claim 4, it is characterized in that the input of the multi-scale feature fusion module guided by the reconstruction feature in step 3 is the denoised image feature and the reconstructed image feature obtained by two feature encoders; first, the denoised image feature and the reconstructed image feature are input into the fusion module, and in the fusion module, the two features are respectively input into different convolutional neural networks with convolution kernels of 3×3 and step sizes of 1, 2, and 2 for downsampling, and denoised image features and reconstructed image features of three scales are obtained respectively, and then the multi-scale reconstructed image features are respectively input into the convolutional neural network with a convolution kernel of 3×3 and a step size of 1 to generate denoised images of each scale. The offset of 9 irregular sampling points at each scale relative to each feature point in the image feature map; according to the multi-scale offset map obtained under the guidance of the reconstructed features, each feature point in the feature map of each scale in the denoised image feature is convolved with 27 sampling points of 9 at each of its three scales to generate a new multi-scale denoised image feature map, which is then input into the next layer of feature fusion; after the last layer of fusion, the small-scale denoised image feature map is upsampled and spliced in turn to finally obtain the denoised image feature map of the original size; it is input into the low-light denoising decoder for feature decoding to obtain the enhanced image, and the decoder structure uses the decoder structure in Transformer. 6. A low-light image enhancement method based on random region hidden reconstruction according to claim 1, characterized in that the reconstruction loss and perceptual loss functions in step 2 and step 3 are L1 and L2 norms; the Adam optimizer is used in the first stage training, wherein the exponential decay rate of the first-order moment estimation β ₁ =0.9, and the exponential decay rate of the second-order moment estimation β ₂ =0.999; the low-light denoising reconstruction model of the first stage is trained for 100 rounds, with an initial learning rate of 0.0006, which linearly decays to 0.0003 after 50 rounds; after the first stage training, the low-light enhancement denoising model of the second stage is trained for 30 rounds using a real low-light dataset, with a learning rate of 0.0002; in order to increase the availability of data, 500 pairs of real low-light datasets are randomly cropped and horizontally flipped during the training process to perform data enhancement to improve the low-light enhancement performance of the second stage task.