WO2021196401A1 - Image reconstruction method and apparatus, electronic device and storage medium - Google Patents

Abstract

The present disclosure relates to an image reconstruction method and apparatus, an electronic device, and a storage medium. The method comprises: acquiring event information of a target scene, the event information being used to represent changes in brightness of the target scene within a first brightness range; performing feature extraction on the event information to obtain a first event feature of the target scene; and performing image reconstruction on the first event feature to obtain a reconstructed image of the target scene, wherein the brightness of the reconstructed image is within a second brightness range, and the second brightness range is higher than the first brightness range. The embodiments of the present disclosure may improve the effect of image reconstruction.

Description

Image reconstruction method and device, electronic equipment and storage medium

This application claims the priority of a Chinese patent application filed with the Chinese Patent Office, the application number is 202010243153.4, and the invention title is "Image reconstruction method and device, electronic equipment and storage medium" on March 31, 2020, the entire content of which is incorporated by reference In this application.

Technical field

The present disclosure relates to the field of computer technology, and in particular to an image reconstruction method and device, electronic equipment and storage medium.

Background technique

Traditional image acquisition equipment can acquire images that conform to people's observation habits, such as RGB images or intensity images. However, limited by its own low dynamic range, the image acquisition device will be underexposed under low light conditions and cannot generate high-quality clear images.

Summary of the invention

The present disclosure proposes a technical solution for image reconstruction.

According to an aspect of the present disclosure, there is provided an image reconstruction method, including: acquiring event information of a target scene, where the event information is used to indicate a brightness change of the target scene within a first brightness range; Performing feature extraction to obtain the first event feature of the target scene; performing image reconstruction on the first event feature to obtain a reconstructed image of the target scene, the brightness of the reconstructed image is within the second brightness range, the The second brightness range is higher than the first brightness range.

In a possible implementation manner, performing image reconstruction on the first event feature to obtain a reconstructed image of the target scene includes: performing image reconstruction on the first event according to the first noise information and the first event feature. Features are enhanced in detail to obtain a second event feature; the first event feature and the second event feature are fused to obtain a fusion feature; the fusion feature is image-reconstructed to obtain a reconstructed image of the target scene.

In a possible implementation manner, the method is implemented by an image processing network, the image processing network includes a first feature extraction network and an image reconstruction network, and the first feature extraction network is used to characterize the event information Extracting, the image reconstruction network is used for image reconstruction of the first event feature, and the method further includes: training the image processing network according to a preset training set, the training set including a plurality of first samples The first sample event information of the scene, the second sample event information of a plurality of second sample scenes, and the sample scene image; wherein, the first sample event information is acquired within a third brightness range, and the second The sample event information is acquired in a fourth brightness range, the sample scene image is acquired in the fourth brightness range, and the fourth brightness range is higher than the third brightness range.

In a possible implementation manner, the image processing network further includes a discrimination network, and the training of the image processing network according to a preset training set includes: combining the first sample event of the first sample scene The information and the second sample event information of the second sample scene are respectively input to the first feature extraction network to obtain the first sample event feature and the second sample event feature; combine the first sample event feature and the The second sample event features are respectively input to the identification network to obtain a first identification result and a second identification result; according to the first identification result and the second identification result, the image processing network is counter-trained.

In a possible implementation manner, the training the image processing network according to a preset training set further includes: inputting the second sample event feature into the image reconstruction network to obtain the image of the second sample scene A first reconstructed image; training the image processing network according to the first reconstructed image of the second sample scene and the sample scene image.

In a possible implementation, the image processing network further includes a detail enhancement network, and the training of the image processing network according to a preset training set further includes: combining the second sample event feature and the third noise The information is input into the detail enhancement network to obtain the fourth sample event feature; the second sample event feature is fused with the fourth sample event feature to obtain the second sample fusion feature; the second sample fusion feature is input to the office The image reconstruction network obtains a third reconstructed image of the second sample scene; training the image processing network according to the first reconstructed image of the second sample scene, the third reconstructed image, and the sample scene image .

In a possible implementation manner, the image processing network further includes a second feature extraction network, and the training of the image processing network according to a preset training set further includes: combining the second sample scene of the second Sample event information and second noise information are input into the second feature extraction network to obtain a third sample event feature; fuse the second sample event feature with the third sample event feature to obtain a first sample fusion feature; The fusion feature of the first sample is input into the identification network to obtain a third identification result; and the image processing network is trained against training according to the first identification result and the third identification result.

In a possible implementation, the training the image processing network according to a preset training set further includes: inputting the first sample fusion feature into the image reconstruction network to obtain the second sample scene Training the image processing network according to the second reconstructed image of the second sample scene and the sample scene image.

In a possible implementation, the image processing network further includes a detail enhancement network, and the training of the image processing network according to a preset training set further includes: fusing features of the first sample and the fourth The noise information is input into the detail enhancement network to obtain the fifth sample event feature; the first sample fusion feature and the fifth sample event feature are fused to obtain the third sample fusion feature; the third sample is fused with features Input the image reconstruction network to obtain the fourth reconstructed image of the second sample scene; train the image according to the second reconstructed image of the second sample scene, the fourth reconstructed image and the sample scene image Deal with the network.

In a possible implementation manner, the training of the image processing network according to the second reconstructed image of the second sample scene, the fourth reconstructed image, and the sample scene image includes: according to the first Determine the overall loss of the image processing network based on the second reconstructed image of the two-sample scene, the fourth reconstructed image, and the sample scene image; determine the gradient information of the image processing network according to the overall loss; The gradient information adjusts the network parameters of the first feature extraction network, the second feature extraction network, the detail enhancement network, and the image reconstruction network, wherein the gradient information of the detail enhancement network is not transmitted to all The second feature extraction network.

According to an aspect of the present disclosure, there is provided an image reconstruction device, including:

The event acquisition module is used to acquire event information of the target scene, where the event information is used to indicate the brightness change of the target scene within the first brightness range; the feature extraction module is used to perform feature extraction on the event information to obtain The first event feature of the target scene; an image reconstruction module for performing image reconstruction on the first event feature to obtain a reconstructed image of the target scene, and the brightness of the reconstructed image is within the second brightness range, so The second brightness range is higher than the first brightness range.

In a possible implementation manner, the image reconstruction module includes: a detail enhancement submodule, configured to perform detail enhancement on the first event feature according to the first noise information and the first event feature to obtain a second Event feature; a fusion sub-module for fusing the first event feature with the second event feature to obtain a fusion feature; a reconstruction sub-module for performing image reconstruction on the fusion feature to obtain the target scene Reconstruct the image.

In a possible implementation manner, the device is implemented by an image processing network, the image processing network includes a first feature extraction network and an image reconstruction network, and the first feature extraction network is used to characterize the event information Extracting, the image reconstruction network is used to perform image reconstruction on the first event feature, and the device further includes:

A training module for training the image processing network according to a preset training set, the training set including first sample event information of multiple first sample scenes, and second sample event information of multiple second sample scenes And sample scene images; wherein the first sample event information is acquired in a third brightness range, the second sample event information is acquired in a fourth brightness range, and the sample scene image is in the Obtained in the fourth brightness range, the fourth brightness range is higher than the third brightness range.

In a possible implementation manner, the image processing network further includes an identification network, and the training module includes: a first extraction submodule, configured to combine the first sample event information of the first sample scene with all the The second sample event information of the second sample scene is respectively input into the first feature extraction network to obtain the first sample event feature and the second sample event feature; the first discrimination sub-module is used to combine the first sample The event feature and the second sample event feature are respectively input to the identification network to obtain the first identification result and the second identification result; the first confrontation training sub-module is used for the first identification result and the second identification result As a result, the image processing network is trained against training.

In a possible implementation, the training module further includes: a first reconstruction submodule, configured to input the second sample event feature into the image reconstruction network to obtain a first reconstruction of the second sample scene Image; a first training sub-module for training the image processing network according to the first reconstructed image of the second sample scene and the sample scene image.

In a possible implementation, the image processing network further includes a detail enhancement network, and the training module further includes: a first enhancement sub-module for inputting the second sample event feature and the third noise information into the The detail enhancement network obtains the fourth sample event feature; the first fusion submodule is used to fuse the second sample event feature with the fourth sample event feature to obtain the second sample fusion feature; the second reconstruction submodule , Used to input the second sample fusion feature into the image reconstruction network to obtain a third reconstructed image of the second sample scene; a second training sub-module for the first reconstruction of the second sample scene The image, the third reconstructed image, and the sample scene image are used to train the image processing network.

In a possible implementation, the image processing network further includes a second feature extraction network, and the training module further includes: a second extraction sub-module configured to combine the second sample event information of the second sample scene And the second noise information is input into the second feature extraction network to obtain the third sample event feature; the second fusion sub-module is used to fuse the second sample event feature with the third sample event feature to obtain the first Sample fusion features; a second discrimination sub-module, used to input the first sample fusion features into the discrimination network to obtain a third discrimination result; a second confrontation training sub-module, used according to the first discrimination result And the third discrimination result, against training the image processing network.

In a possible implementation, the training module further includes: a third reconstruction sub-module, configured to input the first sample fusion feature into the image reconstruction network to obtain the second sample scene of the second sample Reconstructed image; a third training sub-module for training the image processing network according to the second reconstructed image of the second sample scene and the sample scene image.

In a possible implementation, the image processing network further includes a detail enhancement network, and the training module further includes: a second enhancement sub-module for inputting the first sample fusion feature and fourth noise information The detail enhancement network obtains the fifth sample event feature; the third fusion sub-module is used to fuse the first sample fusion feature with the fifth sample event feature to obtain the third sample fusion feature; fourth reconstruction The sub-module is used to input the third sample fusion feature into the image reconstruction network to obtain the fourth reconstructed image of the second sample scene; the fourth training sub-module is used to obtain the fourth reconstructed image of the second sample scene. Training the image processing network with two reconstructed images, the fourth reconstructed image, and the sample scene image.

In a possible implementation manner, the fourth training submodule is configured to: determine the image processing according to the second reconstructed image of the second sample scene, the fourth reconstructed image, and the sample scene image The overall loss of the network; according to the overall loss, determine the gradient information of the image processing network; according to the gradient information, adjust the first feature extraction network, the second feature extraction network, the detail enhancement network, and The network parameters of the image reconstruction network, wherein the gradient information of the detail enhancement network is not transferred to the second feature extraction network.

According to an aspect of the present disclosure, there is provided an electronic device including: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to call the instructions stored in the memory to execute the foregoing method.

According to an aspect of the present disclosure, there is provided a computer-readable storage medium having computer program instructions stored thereon, and the computer program instructions implement the above-mentioned method when executed by a processor.

According to an aspect of the present disclosure, there is provided a computer program including computer readable code, and when the computer readable code runs in an electronic device, a processor in the electronic device executes the above method.

In the embodiment of the present disclosure, it is possible to obtain event information of the target scene in the lower first brightness range; perform feature extraction on the event information to obtain the event feature; perform image reconstruction on the event feature to obtain the target scene in the higher first brightness range. Two reconstructed images in the brightness range, so that high-quality images under normal lighting conditions can be reconstructed through events under dark light conditions, and the effect of image reconstruction is improved.

It should be understood that the above general description and the following detailed description are only exemplary and explanatory, rather than limiting the present disclosure. According to the following detailed description of exemplary embodiments with reference to the accompanying drawings, other features and aspects of the present disclosure will become clear.

Description of the drawings

The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments that conform to the present disclosure, and are used together with the specification to explain the technical solutions of the present disclosure.

Fig. 1 shows a flowchart of an image reconstruction method according to an embodiment of the present disclosure.

Fig. 2 shows a schematic diagram of a processing procedure of network training of an image reconstruction method according to an embodiment of the present disclosure.

Fig. 3 shows a block diagram of an image reconstruction device according to an embodiment of the present disclosure.

Fig. 4 shows a block diagram of an electronic device according to an embodiment of the present disclosure.

Fig. 5 shows a block diagram of an electronic device according to an embodiment of the present disclosure.

Detailed ways

Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the drawings. The same reference numerals in the drawings indicate elements with the same or similar functions. Although various aspects of the embodiments are shown in the drawings, unless otherwise noted, the drawings are not necessarily drawn to scale.

The dedicated word "exemplary" here means "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" need not be construed as being superior or better than other embodiments.

The term "and/or" in this article is only an association relationship describing the associated objects, which means that there can be three relationships, for example, A and/or B, which can mean: A alone exists, A and B exist at the same time, exist alone B these three situations. In addition, the term "at least one" in this document means any one of a plurality of or any combination of at least two of the plurality, for example, including at least one of A, B, and C, may mean including A, Any one or more elements selected in the set formed by B and C.

In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. Those skilled in the art should understand that the present disclosure can also be implemented without certain specific details. In some instances, the methods, means, elements, and circuits well known to those skilled in the art have not been described in detail, so as to highlight the gist of the present disclosure.

In the fields of image shooting, image processing, face recognition, security, etc., it is usually necessary to collect images through an image acquisition device (such as an intensity camera or a camera, etc.). Images collected by an image acquisition device under dark conditions (for example, at night, under light or other dark environments) are prone to underexposure and poor image quality. In this case, a poor quality image can be reconstructed to obtain a high-quality image under normal lighting conditions.

Fig. 1 shows a flowchart of an image reconstruction method according to an embodiment of the present disclosure. As shown in Fig. 1, the method includes:

In step S11, event information of the target scene is acquired, where the event information is used to indicate a brightness change of the target scene within a first brightness range;

In step S12, feature extraction is performed on the event information to obtain the first event feature of the target scene;

In step S13, image reconstruction is performed on the first event feature to obtain a reconstructed image of the target scene, the brightness of the reconstructed image is within a second brightness range, and the second brightness range is higher than the first brightness range. Brightness range.

In a possible implementation, the image reconstruction method can be executed by electronic equipment such as a terminal device or a server, and the terminal device can be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, or a cordless For telephones, personal digital assistants (PDAs), handheld devices, computing devices, vehicle-mounted devices, wearable devices, etc., the method can be implemented by a processor invoking computer-readable instructions stored in a memory. Alternatively, the method can be executed by a server.

In a possible implementation manner, the target scene may be a geographic area including scenes such as buildings, landscapes, people, and vehicles. The target scene may be in a dark light condition (for example, night or other dark environment), and the image of the target scene collected by an image acquisition device (for example, an intensity camera or a camera, etc.) is underexposed and the image quality is poor. In this case, in step S11, an event collection device (such as an event camera) may be used to obtain event information of the target scene within the first brightness range corresponding to the dark light condition, and the event information is used to indicate the target scene The brightness change in the first brightness range. The present disclosure does not limit the specific value of the first brightness range.

In a possible implementation, the event camera can asynchronously record the brightness changes in the scene, and output event data in the form of a stream (event stream). The data unit is as follows:

e _k = (x _k ,y _k ,p _k ,t _k ) (1)

In formula (1), x _k and y _k represent the spatial coordinates _{of the event data e k} at the k-th position in the scene _{, t k} represents the time when the event data e _{k was generated, and p k} ∈{+1,-1} represents the event The polarity of the data e _k. A positive polarity indicates that the brightness is increased, and a negative polarity indicates that the brightness is decreased.

The traditional CNN method can only process regular data in the form of pictures, and cannot be applied to event streams. Therefore, when the target scene is in the first brightness range, the brightness changes of the target scene in one or more preset time periods can be collected by the event collection device to obtain event data, and the polarity of each event data can be determined in the spatial dimension. Perform integration to obtain single-channel or multi-channel event information.

The integration method is as follows:

In formula (2), Φ _τ (t) represents the event information of the event data at the k-th position in the preset time period t _k ∈[t,t+τ]. In this way, by integrating the event data of each position in the scene, a single channel of event information (also called event frame) can be obtained; by integrating the event data of each position in multiple preset time periods, a multi-channel can be obtained. Event information, such as four-channel event information. In order to ensure the consistency of the data range, the event information of each channel can be normalized in the spatial dimension, and the normalized event information can be used as the event information of the target scene. This disclosure does not limit the number of channels for event information.

In a possible implementation manner, feature extraction may be performed on the event information in step S12 to obtain the first event feature of the target scene. The first event feature includes at least information representing the structure of the target scene. The feature of the event information may be extracted by, for example, a convolutional neural network, which may include multiple convolutional layers, multiple residual layers, etc. The present disclosure does not limit the network structure of the convolutional neural network.

In a possible implementation manner, image reconstruction may be performed on the first event feature in step S13 to obtain a reconstructed image of the target scene. The reconstructed image may be, for example, an intensity image, the brightness of the reconstructed image is within a second brightness range corresponding to normal lighting conditions, and the second brightness range is higher than the first brightness range.

In a possible implementation manner, image reconstruction of the first event feature may be performed, for example, by a deconvolutional neural network, which may include multiple deconvolutional layers, multiple residual layers, and convolutional layers. Etc., the present disclosure does not limit the specific value of the second brightness range and the network structure of the deconvolution neural network.

According to the embodiment of the present disclosure, it is possible to obtain event information of the target scene in the lower first brightness range; perform feature extraction on the event information to obtain the event feature; perform image reconstruction on the event feature to obtain the target scene in the higher first brightness range. Two reconstructed images in the brightness range, so that high-quality images under normal lighting conditions can be reconstructed through events under dark light conditions, and the effect of image reconstruction is improved.

In a possible implementation manner, step S13 may include:

Performing detail enhancement on the first event feature according to the first noise information and the first event feature to obtain a second event feature;

Fusing the first event feature and the second event feature to obtain a fusion feature;

Image reconstruction is performed on the fusion feature to obtain a reconstructed image of the target scene.

For example, the event information obtained under dim light conditions may have more noise interference and partial structural information loss. In this case, the first event feature can be enhanced to recover more detailed information.

In a possible implementation manner, random first noise information may be preset, and an additional noise channel is added to the first event feature according to the first noise information. The feature of the first event after the noise channel is added is input into the detail enhancement network for detail enhancement to obtain the feature of the second event. The detail enhancement network may be, for example, a residual network, including a convolutional layer and multiple residual layers. The present disclosure does not limit the method of obtaining the first noise information and the specific network structure of the detail enhancement network.

In a possible implementation, the first event feature and the second event feature can be fused, for example, superimposed, to obtain the fusion feature; the fusion feature is input into the deconvolutional neural network for image reconstruction to obtain the reconstruction of the target scene image.

In this way, the detailed information in the first event feature can be enhanced, and the quality of the reconstructed image can be further improved.

In a possible implementation manner, the image reconstruction method according to the embodiment of the present disclosure may be implemented by an image processing network. The image processing network includes at least a first feature extraction network and an image reconstruction network. The event information is feature extraction, for example, a convolutional neural network; the image reconstruction network is used to perform image reconstruction on the first event feature, for example, a deconvolutional neural network.

It should be understood that the image processing network may adopt other types of networks or models, which can be set by those skilled in the art according to actual conditions, and the present disclosure does not limit this.

Before applying the image processing network, the image processing network can be trained.

In a possible implementation manner, the image reconstruction method according to the embodiment of the present disclosure further includes: training the image processing network according to a preset training set, and the training set includes a first sample of a plurality of first sample scenes. This event information, second sample event information and sample scene images of multiple second sample scenes,

Wherein, the first sample event information is acquired in a third brightness range, the second sample event information is acquired in a fourth brightness range, and the sample scene image is in the fourth brightness range Obtained within, the fourth brightness range is higher than the third brightness range.

For example, a training set may be preset, and the training set includes multiple sample scenes, such as scenes such as buildings, landscapes, people, and vehicles. The sample scenes can be divided into dark light scenes (may be called the first sample scene) and normal lighting scenes (may be called the second sample scene). Each first sample scene includes first sample event information; each second sample scene includes second sample event information and sample scene images. The first sample scene and the second sample scene may be the same or different scenes, which is not limited in the present disclosure.

In a possible implementation manner, when the first sample scene is in the third brightness range corresponding to the dark light condition, the brightness change of the first sample scene can be acquired through an event collection device (for example, an event camera) to obtain The first sample event information in order to serve as the input of the image processing network. The first sample event information includes information representing the overall structure of the first sample scene. The third brightness range may be the same as or different from the aforementioned first brightness range, which is not limited in the present disclosure.

The first sample event information under dark light conditions includes information representing the overall structure of the first sample scene, but lacks intensity information (that is, brightness information of the image). In this case, event information of the second sample scene under normal lighting conditions (may be referred to as second sample event information) can be introduced, so as to learn the intensity information in the second sample event information through the image processing network.

In a possible implementation manner, when the second sample scene is in the fourth brightness range corresponding to the normal lighting condition, the brightness change of the second sample scene can be acquired by the event collection device to obtain the second sample event information. The fourth brightness range is higher than the third brightness range. Wherein, the fourth brightness range may be the same as or different from the aforementioned second brightness range, which is not limited in the present disclosure.

The method for acquiring the first sample event information of the first sample scene and the second sample event information of the second sample scene may be similar to the method for acquiring event information of the target scene, and the description will not be repeated here.

In addition, for the first sample scene under dark light conditions, the image quality of the target scene collected by the image acquisition device is poor and cannot be used as supervision information. In this case, the sample scene image of the second sample scene under normal lighting conditions can be introduced as the supervision information of the image processing network. The sample scene image can be acquired by an image acquisition device (for example, a camera) in a fourth brightness range corresponding to normal lighting conditions.

In this way, the training effect of the image processing network can be improved.

In a possible implementation manner, the image processing network further includes an identification network, and the step of training the image processing network according to a preset training set includes:

The first sample event information of the first sample scene and the second sample event information of the second sample scene are respectively input to the first feature extraction network to obtain the first sample event feature and the second sample event feature;

Input the first sample event feature and the second sample event feature into the authentication network respectively to obtain a first authentication result and a second authentication result;

According to the first identification result and the second identification result, the image processing network is trained against training.

For example, the authentication network in the image processing network is used to authenticate the output result of the first feature extraction network. That is to say, the first feature extraction network can be trained by adversarial training, so that the first feature extraction network can learn between the first sample event information under dark light conditions and the second sample event information under normal lighting conditions. Distribute information together.

In a possible implementation manner, the first sample event information of the first sample scene and the second sample event information of the second sample scene can be separately input into the first feature extraction network for processing, and the first sample is output Event feature and second sample event feature; input the first sample event feature and the second sample event feature into the identification network respectively to obtain the first identification result and the second identification result; according to the first identification result and the second identification result, fight against Training the image processing network.

In the adversarial training process, the first feature extraction network tries to confuse the first sample event feature and the second sample event feature, and the discrimination network tries to distinguish the first sample event feature from the second sample event feature. The two oppose each other and promote each other. .

In this way, the first feature extraction network can be forced to extract the common distribution domain between the feature domain under normal lighting conditions and the feature domain under dim light conditions, so that the first sample event feature under dim light conditions has normal lighting conditions. The event information distribution characteristics of the second sample under normal light conditions have the distribution characteristics of event information under dim light conditions. That is to say, the first feature extraction network is suitable for feature extraction of two differently distributed data at the same time through the way of domain adaptation. The present disclosure does not limit the selection of the loss function for the confrontation training.

In this way, the first feature extraction network can better extract event features under dark light, and the accuracy of the first feature extraction network can be improved, so that high-quality image reconstruction can be realized by using event information under dark light.

In a possible implementation, the step of training the image processing network according to a preset training set further includes:

Inputting the feature of the second sample event into the image reconstruction network to obtain a first reconstructed image of the second sample scene;

Training the image processing network according to the first reconstructed image of the second sample scene and the sample scene image.

For example, after the confrontation training, the second sample event features extracted by the first feature extraction network have the distribution characteristics of event information under dark light conditions, and the corresponding second sample event information has supervision information (that is, , The sample scene image under normal lighting conditions).

In a possible implementation, the second sample event feature can be input into the image reconstruction network for processing, and output the first reconstructed image of the second sample scene; according to the first reconstructed image of the second sample scene and the sample scene image The difference between the two can determine the network loss of the first feature extraction network and the image reconstruction network, such as L1 loss; further, the network parameters of the first feature extraction network and the image reconstruction network can be adjusted inversely according to the network loss to realize the first feature Extraction network and image reconstruction network training.

In the actual training process, alternate training can be carried out. That is, in each round of iterative process, the network parameters of the identification network are adjusted in the reverse direction according to the loss of the counter network. Then according to the network loss of the first feature extraction network and image reconstruction network, reversely adjust the network parameters of the first feature extraction network and image reconstruction network. In this training, the output of the identification network will still be obtained as the guidance information, but the identification will not be updated. The parameters of the network. In this way, after multiple rounds of iteration, the trained image processing network can be obtained when the training conditions (such as network convergence) are met.

In this way, the training process of the entire image processing network can be realized, and a high-precision image processing network can be obtained.

In a possible implementation manner, the image processing network further includes a second feature extraction network, and the step of training the image processing network according to a preset training set further includes:

Input second sample event information and second noise information of the second sample scene into the second feature extraction network to obtain third sample event features;

Fusing the second sample event feature with the third sample event feature to obtain a first sample fusion feature;

Input the fusion feature of the first sample into the authentication network to obtain a third authentication result;

According to the first identification result and the third identification result, the image processing network is trained against training.

For example, the first sample event information under dim light conditions may have certain noise interference, while the second sample event information under normal lighting conditions has low noise. In this case, additional noise channels can be introduced for the second sample event information to improve the generalization of the network.

In a possible implementation, the image processing network further includes a second feature extraction network, for example, a convolutional image processing network, which includes multiple convolutional layers and multiple residual layers. The network structure is not restricted.

In a possible implementation manner, random second noise information may be preset, and a noise channel is added to the second sample event information according to the second noise information. The second sample event information after adding the noise channel is input into the second feature extraction network for feature extraction, and the third sample event feature is output; the second sample event feature is fused with the third sample event feature to obtain the first Sample fusion features. In this way, the feature enhancement of the second sample event feature can be achieved.

In a possible implementation manner, the first sample fusion feature is input into the identification network to obtain a third identification result; further, the image processing network is opposed to training based on the first identification result and the third identification result. The specific process of confrontation training will not be repeated.

In this way, the accuracy of the first feature extraction network can be further improved.

In a possible implementation manner, the image processing network further includes a second feature extraction network, and the step of training the image processing network according to a preset training set further includes:

Input the first sample fusion feature into the image reconstruction network to obtain a second reconstructed image of the second sample scene;

Training the image processing network according to the second reconstructed image of the second sample scene and the sample scene image.

For example, after the confrontation training, the first sample fusion features extracted by the first feature extraction network and the second feature extraction network have the characteristics of the distribution of event information under dark light conditions, and the corresponding second sample event The information has supervision information (that is, the sample scene image under normal lighting conditions).

In a possible implementation, the fusion feature of the first sample can be input to the image reconstruction network for processing, and the second reconstructed image of the second sample scene is output; the second reconstructed image and the sample scene image of the second sample scene are output. The difference between the network loss of the first feature extraction network, the second feature extraction network, and the image reconstruction network can be determined, such as L1 loss; further, the first feature extraction network and the second feature extraction can be adjusted inversely according to the network loss The network parameters of the network and the image reconstruction network realize the training of the first feature extraction network, the second feature extraction network and the image reconstruction network.

In the actual training process, alternate training can also be performed. That is, in each round of iterative process, the network parameters of the identification network are adjusted backward according to the loss of the counter network; then the network parameters of the first feature extraction network, the second feature extraction network, and the image reconstruction network are adjusted backwards according to the network loss of the first feature extraction network, the second feature extraction network, and the image reconstruction network. The network parameters of the feature extraction network, the second feature extraction network and the image reconstruction network will still receive the output of the authentication network as guidance information during this training, but the parameters of the authentication network will not be updated. In this way, after multiple rounds of iteration, the trained image processing network can be obtained when the training conditions (such as network convergence) are met.

In this way, the training process of the entire image processing network can be realized, and a high-precision image processing network can be obtained.

In a possible implementation manner, the image processing network further includes a detail enhancement network, and the step of training the image processing network according to a preset training set may further include:

Input the second sample event feature and the third noise information into the detail enhancement network to obtain the fourth sample event feature;

Fusing the second sample event feature with the fourth sample event feature to obtain a second sample fusion feature;

Inputting the second sample fusion feature into the image reconstruction network to obtain a third reconstructed image of the second sample scene;

Training the image processing network according to the first reconstructed image of the second sample scene, the third reconstructed image, and the sample scene image.

For example, a detail enhancement network can be introduced to enhance the details of event features, so as to recover more image detail information (such as local structural information). The detail enhancement network may be, for example, a residual network, including a convolutional layer and multiple residual layers, and the present disclosure does not limit the network structure of the detail enhancement network.

In a possible implementation manner, in the case where the second feature extraction network is not introduced, the second sample event feature can be directly used for detail enhancement. Random third noise information can be preset, and a noise channel is added to the second sample event feature according to the third noise information. The second sample event feature after adding the noise channel is input into the detail enhancement network for processing to obtain the fourth sample event feature; the second sample event feature is fused with the fourth sample event feature to obtain the second sample fusion feature; The two-sample fusion feature is input into the image reconstruction network to obtain a third reconstructed image of the second sample scene.

In a possible implementation manner, the image processing network is trained according to the first reconstructed image, the third reconstructed image, and the sample scene image of the sample scene.

Among them, according to the difference between the third reconstructed image and the sample scene image, the first loss of the first feature extraction network, the detail enhancement network, and the image reconstruction network can be determined; according to the difference between the third reconstructed image and the sample scene image, And the difference between the first reconstructed image and the sample scene image can determine the second loss of the first feature extraction network, the detail enhancement network, and the image reconstruction network. The second loss can ensure that the quality of the third reconstructed image after the detail enhancement is introduced is better than the quality of the first reconstructed image when the detail enhancement is not introduced, ensuring that the detail enhancement network can play an expected role.

In a possible implementation manner, the total loss of the first feature extraction network, the detail enhancement network, and the image reconstruction network can be determined according to the first loss and the second loss, for example, the weighted sum of the first loss and the second loss is determined as The overall loss; further, the network parameters of the first feature extraction network, the detail enhancement network, and the image reconstruction network can be adjusted inversely according to the overall loss to realize the training of the first feature extraction network, the detail enhancement network, and the image reconstruction network.

In the actual training process, alternate training can also be performed. That is, in each iteration process, the identification network is trained against the training; the first feature extraction network, the detail enhancement network and the image reconstruction network are trained again, and the output of the identification network is used as guidance information, but the parameters of the identification network are not updated. After multiple rounds of iteration, the trained image processing network can be obtained if the training conditions (such as network convergence) are met.

In this way, the details of the reconstructed image can be enhanced, and the quality of the reconstructed image obtained by the trained image processing network can be further improved.

In a possible implementation, the step of training the image processing network according to a preset training set may further include:

Input the first sample fusion feature and the fourth noise information into the detail enhancement network to obtain the fifth sample event feature;

Fuse the first sample fusion feature with the fifth sample event feature to obtain a third sample fusion feature;

Inputting the third sample fusion feature into the image reconstruction network to obtain a fourth reconstructed image of the second sample scene;

Training the image processing network according to the second reconstructed image of the second sample scene, the fourth reconstructed image, and the sample scene image.

For example, in the case where the second feature extraction network has been introduced, the first sample fusion feature can be used for detail enhancement. Random fourth noise information can be preset, and a noise channel is added to the fusion feature of the first sample according to the fourth noise information. The fusion feature of the first sample after adding the noise channel is input into the detail enhancement network for processing to obtain the event feature of the fifth sample; the fusion feature of the first sample is fused with the event feature of the fifth sample to obtain the fusion feature of the third sample; The third sample fusion feature is input into the image reconstruction network to obtain a fourth reconstructed image of the second sample scene.

In a possible implementation manner, an image processing network is trained according to the second reconstructed image of the second sample scene, the fourth reconstructed image, and the sample scene image. This step can include:

Determine the overall loss of the image processing network according to the second reconstructed image of the second sample scene, the fourth reconstructed image, and the sample scene image;

Determine the gradient information of the image processing network according to the overall loss;

Adjusting the network parameters of the first feature extraction network, the second feature extraction network, the detail enhancement network, and the image reconstruction network according to the gradient information,

Wherein, the gradient information of the detail enhancement network is not transmitted to the second feature extraction network.

For example, according to the difference between the fourth reconstructed image and the sample scene image, the third loss of the first feature extraction network, the second feature extraction network, the detail enhancement network, and the image reconstruction network can be determined; according to the fourth reconstructed image and The difference between the sample scene images and the difference between the second reconstructed image and the sample scene image can determine the fourth loss of the first feature extraction network, the second feature extraction network, the detail enhancement network, and the image reconstruction network. The fourth loss can ensure that the quality of the fourth reconstructed image after the detail enhancement is introduced is better than the quality of the second reconstructed image when the detail enhancement is not introduced, and ensures that the detail enhancement network can play an expected role.

In a possible implementation, the total loss of the first feature extraction network, the second feature extraction network, the detail enhancement network, and the image reconstruction network can be determined according to the third loss and the fourth loss, for example, the third loss and the fourth loss The weighted sum of the loss is determined as the overall loss; according to the overall loss, the gradient information of the first feature extraction network, the second feature extraction network, the detail enhancement network, and the image reconstruction network can be determined. Two feature extraction network, detail enhancement network and image reconstruction network transfer the gradient information in reverse, so as to adjust the network parameters of the first feature extraction network, second feature extraction network, detail enhancement network and image reconstruction network to realize the first feature extraction Network, second feature extraction network, detail enhancement network and image reconstruction network training.

In a possible implementation, since the input of the second feature extraction network and the detail enhancement network both add noise channels, in order to reduce the impact of the early training stage on the learning effect, when the gradient information is transmitted backward, the details are enhanced Stop gradient transfer between the network and the second feature extraction network, thereby reducing the mutual interference between the detail enhancement network and the second feature extraction network, effectively reducing the loop in the information flow, and reducing the probability of mode collapse.

In the actual training process, alternate training can also be performed. That is, during each iteration, the discriminating network is trained against training. Retrain the first feature extraction network, the second feature extraction network, the detail enhancement network, and the image reconstruction network. The output of the authentication network is used as guidance information, but the parameters of the authentication network are not updated. After multiple rounds of iteration, the trained image processing network can be obtained if the training conditions (such as network convergence) are met.

In this way, the details of the reconstructed image can be enhanced, and the quality of the reconstructed image obtained by the trained image processing network can be further improved.

Fig. 2 shows a schematic diagram of a processing procedure of network training of an image reconstruction method according to an embodiment of the present disclosure. 2, the image processing network according to the disclosed embodiments of the present embodiment comprises a first feature extraction network E _C, the second feature extraction network E _P, D network authentication, detail enhancement and image reconstruction network T _e network R.

In an example, for any set of the first sample and the second sample scenario scenario, the event may be the first sample in the low light condition information input of the first feature extraction network 21 E _C processing, the output of the first sample event Feature X _LE ; input the second sample event information 22 under normal lighting conditions into the first feature extraction network E _C that shares the parameters for processing, and output the second sample event feature X _C ; for the second sample event information under normal lighting conditions adding noise information 22 23, wherein the second input parameter is the shared network E _P extraction processing, the output of the third sample event wherein X _p; a second event wherein X _C sample and the third sample event superimposed wherein X _p, The first sample fusion feature X _{DE is obtained} ; the first sample event feature X _LE and the first sample fusion feature X _DE are respectively input into the identification network D for identification, and respective identification results (not shown) are obtained.

In the example, the discrimination network D is trained against the discrimination result according to the discrimination result. The network loss L _{D is} expressed as follows:

和

分别表示第一样本事件特征X _LE和第一样本融合特征X _DE对应的损失。 In formula (3),

with

Respectively represent the loss corresponding to the first sample event feature X _LE and the first sample fusion feature X _DE.

同时，对第一样本融合特征X _DE添加噪声信息24后，输入细节增强网络T _e中，输出第五样本事件特征Δy；将第一样本融合特征X _DE与第五样本事件特征Δy融合后，输入图像重建网络R中，输出第四重建图像

In the example, the first sample fusion feature X _{DE is} input into the image reconstruction network R, and the second reconstructed image is output

At the same time, after adding noise information 24 to the first sample fusion feature X _DE , it is input into the detail enhancement network _Te , and the fifth sample event feature Δy is output; the first sample fusion feature X _{DE is} fused with the fifth sample event feature Δy After the input image reconstruction network R, output the fourth reconstructed image

第四重建图像

及所述样本场景图像y ^g(未示出)，可确定第一特征提取网络E _C、第二特征提取网络E _P、细节增强网络T _e及图像重建网络R的总体损失L _R(也可称为重建损失)，表示如下： In the example, according to the second reconstructed image

Fourth reconstructed image

The sample image and scene y ^g (not shown), may determine the first feature extraction network E _C, the second feature extraction network E _P, T _e and detail enhancement network overall loss R L _R image reconstruction network (also Called reconstruction loss), expressed as follows:

表示亮度重建损失，可以为第四重建图像

与所述样本场景图像y ^g之间的L1损失，以及第二重建图像

与所述样本场景图像y ^g之间的L1损失的和。L _t(Δy,X _p)表示细节增强网络的残差损失，可以为Δy与-X _p之间的L1损失(表示为||Δy-(-X _p)|| ₁)。

表示排名损失，可以为第四重建图像

与所述样本场景图像y ^g之间的L1损失，以及第二重建图像

与所述样本场景图像y ^g之间的L1损失的差。β和γ表示超参数项，本领域技术人员可根据实际情况设置。 In formula (4),

Indicates the loss of brightness reconstruction, which can be the fourth reconstructed image

L1 loss with the sample scene image y ^g , and the second reconstructed image

The sum of L1 losses with the sample scene image y ^g. L _t (Δy,X _p ) represents the residual loss of the detail enhancement network, which can be _{the L1 loss between Δy and -X p} (expressed as ||Δy-(-X _p )|| ₁ ).

Represents the ranking loss, which can be the fourth reconstructed image

L1 loss with the sample scene image y ^g , and the second reconstructed image

The difference in L1 loss with the sample scene image y ^g. β and γ represent hyperparameter items, which can be set by those skilled in the art according to actual conditions.

Wherein, L _R is used to ensure that the first network to recover the correct image, the second network to ensure the accuracy of detail enhancement, the effect of the third reconstruction for the network to ensure that after introduction of the detail enhancement network T _e Better, so that the detail enhancement network _Te can really play the role of detail enhancement.

In an example, the overall optimization goal of the image processing network according to an embodiment of the present disclosure can be expressed as follows:

分别表示用于第一特征提取网络E _C、第二特征提取网络E _P、图像重建网络R及细节增强网络T _e的参数；θ _D表示鉴别网络D的参数；α是相应的超参数权重，本领域技术人员可根据实际情况设置。根据本公开实施例，可使用对抗式训练交替优化这两类参数，可例如采用随机批处理梯度下降的方式进行训练，本公开对此不作限制。经训练后，可得到高精度的图像处理网络。 In formula (5),

Respectively, a first feature extraction network for E _C, the second feature extraction network E _P, R and detail image reconstruction enhanced network parameters of the network of T _{e; D} [theta] D represents the network authentication parameter; [alpha] is hyper-parameters corresponding weights, Those skilled in the art can set it according to the actual situation. According to the embodiments of the present disclosure, adversarial training can be used to alternately optimize the two types of parameters, and training can be performed, for example, in a random batch gradient descent method, which is not limited in the present disclosure. After training, a high-precision image processing network can be obtained.

According to the image reconstruction method of the embodiment of the present disclosure, by combining the domain adaptive method with the event camera, image reconstruction is performed using event information under dark light conditions to obtain high-quality images under normal lighting conditions, which improves the effect of image reconstruction. In the training process, this method does not need to perform supervised training on intensity images under dark light, realizes an unsupervised network framework, and reduces the difficulty of data set construction. This method enhances the dark light distribution domain in the event feature through the detail enhancement network, reduces the noise interference, enhances the local details, and improves the effect of image reconstruction and training.

The network framework of the image reconstruction method according to the embodiments of the present disclosure does not depend on event information, and is also applicable to other tasks based on domain adaptation methods, such as image style transformation and semantic segmentation domain adaptation. Just change the corresponding input data and replace the image reconstruction network with the network structure corresponding to the respective tasks.

The image reconstruction method according to the embodiment of the present disclosure can be applied to the fields of image shooting, image processing, face recognition, security, etc., to realize image reconstruction under dark light conditions.

For example, the shooting system of electronic devices (such as smart phones) using related technologies is based on intensity cameras, which cannot be imaged in low light conditions. Using flash as an aid to take photos or record videos will bring about a great increase in energy consumption, and The harsh light of the flash is very unfriendly to the people in the scene. The highly dynamic event camera does not require additional light source assistance, and the energy consumption is very low. The event camera can be set to acquire event information under dark light conditions, and through the image reconstruction method of the embodiments of the present disclosure, a clear image is generated according to the event information, thereby realizing image shooting under dark light conditions.

For example, the image reconstruction method of the embodiment of the present disclosure can be used as an upstream algorithm of various image processing algorithms. Image processing tasks such as face recognition, object detection, and semantic segmentation will fail due to the inability to obtain high-quality intensity images under low light conditions. This image reconstruction method can reconstruct an intensity image under dark light through event information under dark light conditions, so that the above algorithm can continue to be applied.

For example, a large number of intensity cameras are used in the security field of cities, and there will be many blind spots that cannot be clearly detected in shadow areas and dark light conditions. The event camera can be set to obtain event information under dark light conditions, and through the image reconstruction method of the embodiment of the present disclosure, a clear image is generated according to the event information, thereby improving the effect of security detection and ensuring urban safety.

It can be understood that the various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle and logic. The length is limited, and the details of this disclosure will not be repeated. Those skilled in the art can understand that, in the above method of the specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

In addition, the present disclosure also provides image reconstruction devices, electronic equipment, computer-readable storage media, and programs, all of which can be used to implement any of the image reconstruction methods provided in the present disclosure. For the corresponding technical solutions and descriptions, refer to the corresponding records in the method section. ,No longer.

Fig. 3 shows a block diagram of an image reconstruction device according to an embodiment of the present disclosure. As shown in Fig. 3, the device includes:

The event obtaining module 31 is configured to obtain event information of a target scene, where the event information is used to indicate a brightness change of the target scene within a first brightness range;

The feature extraction module 32 is configured to perform feature extraction on the event information to obtain the first event feature of the target scene;

The image reconstruction module 33 is configured to perform image reconstruction on the first event feature to obtain a reconstructed image of the target scene, the brightness of the reconstructed image is within a second brightness range, and the second brightness range is higher than the The first brightness range.

In a possible implementation manner, the image reconstruction module includes: a detail enhancement submodule, configured to perform detail enhancement on the first event feature according to the first noise information and the first event feature to obtain a second Event feature; a fusion sub-module for fusing the first event feature with the second event feature to obtain a fusion feature; a reconstruction sub-module for performing image reconstruction on the fusion feature to obtain the target scene Reconstruct the image.

In a possible implementation manner, the device is implemented by an image processing network, the image processing network includes a first feature extraction network and an image reconstruction network, and the first feature extraction network is used to characterize the event information Extracting, the image reconstruction network is used to perform image reconstruction on the first event feature, and the device further includes:

A training module for training the image processing network according to a preset training set, the training set including first sample event information of multiple first sample scenes, and second sample event information of multiple second sample scenes And sample scene images; wherein the first sample event information is acquired in a third brightness range, the second sample event information is acquired in a fourth brightness range, and the sample scene image is in the Obtained in the fourth brightness range, the fourth brightness range is higher than the third brightness range.

In a possible implementation manner, the image processing network further includes an identification network, and the training module includes: a first extraction submodule, configured to combine the first sample event information of the first sample scene with all the The second sample event information of the second sample scene is respectively input into the first feature extraction network to obtain the first sample event feature and the second sample event feature; the first discrimination sub-module is used to combine the first sample The event feature and the second sample event feature are respectively input to the identification network to obtain the first identification result and the second identification result; the first confrontation training sub-module is used for the first identification result and the second identification result As a result, the image processing network is trained against training.

In a possible implementation, the training module further includes: a first reconstruction submodule, configured to input the second sample event feature into the image reconstruction network to obtain a first reconstruction of the second sample scene Image; a first training sub-module for training the image processing network according to the first reconstructed image of the second sample scene and the sample scene image.

In a possible implementation, the image processing network further includes a detail enhancement network, and the training module further includes: a first enhancement sub-module for inputting the second sample event feature and the third noise information into the The detail enhancement network obtains the fourth sample event feature; the first fusion submodule is used to fuse the second sample event feature with the fourth sample event feature to obtain the second sample fusion feature; the second reconstruction submodule , Used to input the second sample fusion feature into the image reconstruction network to obtain a third reconstructed image of the second sample scene; a second training sub-module for the first reconstruction of the second sample scene The image, the third reconstructed image, and the sample scene image are used to train the image processing network.

In a possible implementation, the image processing network further includes a second feature extraction network, and the training module further includes: a second extraction sub-module configured to combine the second sample event information of the second sample scene And the second noise information is input into the second feature extraction network to obtain the third sample event feature; the second fusion sub-module is used to fuse the second sample event feature with the third sample event feature to obtain the first Sample fusion features; a second discrimination sub-module, used to input the first sample fusion features into the discrimination network to obtain a third discrimination result; a second confrontation training sub-module, used according to the first discrimination result And the third discrimination result, against training the image processing network.

In a possible implementation, the training module further includes: a third reconstruction sub-module, configured to input the first sample fusion feature into the image reconstruction network to obtain the second sample scene of the second sample Reconstructed image; a third training sub-module for training the image processing network according to the second reconstructed image of the second sample scene and the sample scene image.

In a possible implementation, the image processing network further includes a detail enhancement network, and the training module further includes: a second enhancement sub-module for inputting the first sample fusion feature and fourth noise information The detail enhancement network obtains the fifth sample event feature; the third fusion sub-module is used to fuse the first sample fusion feature with the fifth sample event feature to obtain the third sample fusion feature; fourth reconstruction The sub-module is used to input the third sample fusion feature into the image reconstruction network to obtain the fourth reconstructed image of the second sample scene; the fourth training sub-module is used to obtain the fourth reconstructed image of the second sample scene. Training the image processing network with two reconstructed images, the fourth reconstructed image, and the sample scene image.

In a possible implementation manner, the fourth training submodule is configured to: determine the image processing according to the second reconstructed image of the second sample scene, the fourth reconstructed image, and the sample scene image The overall loss of the network; according to the overall loss, determine the gradient information of the image processing network; according to the gradient information, adjust the first feature extraction network, the second feature extraction network, the detail enhancement network, and The network parameters of the image reconstruction network, wherein the gradient information of the detail enhancement network is not transferred to the second feature extraction network.

In some embodiments, the functions or modules contained in the device provided in the embodiments of the present disclosure can be used to execute the methods described in the above method embodiments. For specific implementation, refer to the description of the above method embodiments. For brevity, here No longer.

The embodiments of the present disclosure also provide a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above-mentioned method when executed by a processor. The computer-readable storage medium may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium.

An embodiment of the present disclosure also proposes an electronic device, including: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

The embodiments of the present disclosure also provide a computer program product, including computer-readable code. When the computer-readable code runs on the device, the processor in the device executes the image reconstruction method provided by any of the above embodiments. instruction.

The embodiments of the present disclosure also provide another computer program product for storing computer-readable instructions, which when executed, cause the computer to perform the operations of the image reconstruction method provided by any of the foregoing embodiments.

The electronic device can be provided as a terminal, server or other form of device.

FIG. 4 shows a block diagram of an electronic device 800 according to an embodiment of the present disclosure. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcasting terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, and other terminals.

4, the electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812, and a sensor component 814 , And communication component 816.

The processing component 802 generally controls the overall operations of the electronic device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the foregoing method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

The memory 804 is configured to store various types of data to support operations in the electronic device 800. Examples of these data include instructions for any application or method to operate on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable and Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic Disk or Optical Disk.

The power supply component 806 provides power for various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with the generation, management, and distribution of power for the electronic device 800.

The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, sliding, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure related to the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and/or a rear camera. When the electronic device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal may be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 further includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and a peripheral interface module. The above-mentioned peripheral interface module may be a keyboard, a click wheel, a button, and the like. These buttons may include, but are not limited to: home button, volume button, start button, and lock button.

The sensor component 814 includes one or more sensors for providing the electronic device 800 with various aspects of state evaluation. For example, the sensor component 814 can detect the on/off status of the electronic device 800 and the relative positioning of the components. For example, the component is the display and the keypad of the electronic device 800. The sensor component 814 can also detect the electronic device 800 or the electronic device 800. The position of the component changes, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration/deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor component 814 may include a proximity sensor configured to detect the presence of nearby objects when there is no physical contact. The sensor component 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor component 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, 2G, or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field-available A programmable gate array (FPGA), controller, microcontroller, microprocessor, or other electronic components are implemented to implement the above methods.

In an exemplary embodiment, there is also provided a non-volatile computer-readable storage medium, such as the memory 804 including computer program instructions, which can be executed by the processor 820 of the electronic device 800 to complete the foregoing method.

FIG. 5 shows a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server. 5, the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932, for storing instructions executable by the processing component 1922, such as application programs. The application program stored in the memory 1932 may include one or more modules each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above-described methods.

The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to the network, and an input output (I/O) interface 1958 . The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as Windows Server ^TM , Mac OS X ^TM , Unix ^TM , Linux ^TM , FreeBSD ^TM or the like.

In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as the memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to complete the foregoing method.

The present disclosure may be a system, method and/or computer program product. The computer program product may include a computer-readable storage medium loaded with computer-readable program instructions for enabling a processor to implement various aspects of the present disclosure.

The computer-readable storage medium may be a tangible device that can hold and store instructions used by the instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (non-exhaustive list) include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) Or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, such as a printer with instructions stored thereon The protruding structure in the hole card or the groove, and any suitable combination of the above. The computer-readable storage medium used here is not interpreted as the instantaneous signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (for example, light pulses through fiber optic cables), or through wires Transmission of electrical signals.

The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to various computing/processing devices, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. The network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network, and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing/processing device .

The computer program instructions used to perform the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or in one or more programming languages. Source code or object code written in any combination, the programming language includes object-oriented programming languages such as Smalltalk, C++, etc., and conventional procedural programming languages such as "C" language or similar programming languages. Computer-readable program instructions can be executed entirely on the user's computer, partly on the user's computer, executed as a stand-alone software package, partly on the user's computer and partly executed on a remote computer, or entirely on the remote computer or server implement. In the case of a remote computer, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to access the Internet). connect). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), can be customized by using the status information of the computer-readable program instructions. The computer-readable program instructions are executed to realize various aspects of the present disclosure.

Here, various aspects of the present disclosure are described with reference to flowcharts and/or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and/or block diagrams, and combinations of blocks in the flowcharts and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine that makes these instructions when executed by the processor of the computer or other programmable data processing device , A device that implements the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams is produced. It is also possible to store these computer-readable program instructions in a computer-readable storage medium. These instructions make computers, programmable data processing apparatuses, and/or other devices work in a specific manner, so that the computer-readable medium storing the instructions includes An article of manufacture, which includes instructions for implementing various aspects of the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams.

It is also possible to load computer-readable program instructions onto a computer, other programmable data processing device, or other equipment, so that a series of operation steps are executed on the computer, other programmable data processing device, or other equipment to produce a computer-implemented process , So that the instructions executed on the computer, other programmable data processing apparatus, or other equipment realize the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams.

The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions, and operations of the system, method, and computer program product according to multiple embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, program segment, or part of an instruction, and the module, program segment, or part of an instruction contains one or more components for realizing the specified logical function. Executable instructions. In some alternative implementations, the functions marked in the block may also occur in a different order from the order marked in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, or they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and/or flowchart, and the combination of the blocks in the block diagram and/or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or actions Or it can be realized by a combination of dedicated hardware and computer instructions.

The computer program product can be specifically implemented by hardware, software, or a combination thereof. In an optional embodiment, the computer program product is specifically embodied as a computer storage medium. In another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc. Wait.

Without violating logic, different embodiments of the present disclosure can be combined with each other, and the description of different embodiments is emphasized. For the part of the description, reference may be made to the records of other embodiments.

The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Without departing from the scope and spirit of the illustrated embodiments, many modifications and changes are obvious to those of ordinary skill in the art. The choice of terms used herein is intended to best explain the principles, practical applications, or improvements to technologies in the market of the embodiments, or to enable other ordinary skilled in the art to understand the embodiments disclosed herein.

Claims (23)

An image reconstruction method, including: Acquiring event information of a target scene, where the event information is used to indicate a brightness change of the target scene within a first brightness range; Performing feature extraction on the event information to obtain the first event feature of the target scene; Image reconstruction is performed on the first event feature to obtain a reconstructed image of the target scene, the brightness of the reconstructed image is within a second brightness range, and the second brightness range is higher than the first brightness range. The method according to claim 1, wherein performing image reconstruction on the first event feature to obtain a reconstructed image of the target scene comprises: Performing detail enhancement on the first event feature according to the first noise information and the first event feature to obtain a second event feature; Fusing the first event feature and the second event feature to obtain a fusion feature; Image reconstruction is performed on the fusion feature to obtain a reconstructed image of the target scene. The method according to claim 1 or 2, wherein the method is implemented by an image processing network, and the image processing network includes a first feature extraction network and an image reconstruction network, and the first feature extraction network is used to compare Performing feature extraction on the event information, and the image reconstruction network is used to perform image reconstruction on the first event feature, The method further includes: training the image processing network according to a preset training set, the training set including first sample event information of a plurality of first sample scenes, and second sample events of a plurality of second sample scenes Information and sample scene images; Wherein, the first sample event information is acquired in a third brightness range, the second sample event information is acquired in a fourth brightness range, and the sample scene image is in the fourth brightness range Obtained within, the fourth brightness range is higher than the third brightness range. The method according to claim 3, wherein the image processing network further comprises a discrimination network, and the training of the image processing network according to a preset training set comprises: The first sample event information of the first sample scene and the second sample event information of the second sample scene are respectively input to the first feature extraction network to obtain the first sample event feature and the second sample event feature; Input the first sample event feature and the second sample event feature into the authentication network respectively to obtain a first authentication result and a second authentication result; According to the first identification result and the second identification result, the image processing network is trained against training. The method according to claim 4, wherein said training said image processing network according to a preset training set, further comprising: Inputting the feature of the second sample event into the image reconstruction network to obtain a first reconstructed image of the second sample scene; Training the image processing network according to the first reconstructed image of the second sample scene and the sample scene image. The method according to claim 5, wherein the image processing network further comprises a detail enhancement network, and the training of the image processing network according to a preset training set further comprises: Input the second sample event feature and the third noise information into the detail enhancement network to obtain the fourth sample event feature; Fusing the second sample event feature with the fourth sample event feature to obtain a second sample fusion feature; Inputting the second sample fusion feature into the image reconstruction network to obtain a third reconstructed image of the second sample scene; Training the image processing network according to the first reconstructed image of the second sample scene, the third reconstructed image, and the sample scene image. The method according to claim 4, wherein the image processing network further comprises a second feature extraction network, and the training of the image processing network according to a preset training set further comprises: Inputting the second sample event information and the second noise information of the second sample scene into the second feature extraction network to obtain the third sample event feature; Fusing the second sample event feature with the third sample event feature to obtain a first sample fusion feature; Input the fusion feature of the first sample into the authentication network to obtain a third authentication result; According to the first identification result and the third identification result, the image processing network is trained against training. The method according to claim 7, wherein said training said image processing network according to a preset training set, further comprising: Input the first sample fusion feature into the image reconstruction network to obtain a second reconstructed image of the second sample scene; Training the image processing network according to the second reconstructed image of the second sample scene and the sample scene image. The method according to claim 8, wherein the image processing network further comprises a detail enhancement network, and the training of the image processing network according to a preset training set further comprises: Input the first sample fusion feature and the fourth noise information into the detail enhancement network to obtain the fifth sample event feature; Fuse the first sample fusion feature with the fifth sample event feature to obtain a third sample fusion feature; Inputting the third sample fusion feature into the image reconstruction network to obtain a fourth reconstructed image of the second sample scene; Training the image processing network according to the second reconstructed image of the second sample scene, the fourth reconstructed image, and the sample scene image. The method according to claim 9, wherein the training the image processing network according to the second reconstructed image of the second sample scene, the fourth reconstructed image, and the sample scene image comprises: Determine the overall loss of the image processing network according to the second reconstructed image of the second sample scene, the fourth reconstructed image, and the sample scene image; Determine the gradient information of the image processing network according to the overall loss; Adjusting the network parameters of the first feature extraction network, the second feature extraction network, the detail enhancement network, and the image reconstruction network according to the gradient information, Wherein, the gradient information of the detail enhancement network is not transmitted to the second feature extraction network. An image reconstruction device includes: An event acquisition module, configured to acquire event information of a target scene, where the event information is used to indicate a brightness change of the target scene within a first brightness range; A feature extraction module, configured to perform feature extraction on the event information to obtain the first event feature of the target scene; The image reconstruction module is configured to perform image reconstruction on the first event feature to obtain a reconstructed image of the target scene, the brightness of the reconstructed image is within a second brightness range, and the second brightness range is higher than the first brightness range. A range of brightness. The apparatus according to claim 11, wherein the image reconstruction module comprises: The detail enhancement sub-module is configured to perform detail enhancement on the first event feature according to the first noise information and the first event feature to obtain a second event feature; A fusion sub-module for fusing the first event feature with the second event feature to obtain a fusion feature; The reconstruction sub-module is used to perform image reconstruction on the fusion feature to obtain a reconstructed image of the target scene. The device according to claim 11 or 12, wherein the device is implemented by an image processing network, the image processing network comprising a first feature extraction network and an image reconstruction network, and the first feature extraction network is used to compare Performing feature extraction on the event information, and the image reconstruction network is used to perform image reconstruction on the first event feature, The device also includes: A training module for training the image processing network according to a preset training set, the training set including first sample event information of multiple first sample scenes, and second sample event information of multiple second sample scenes And sample scene images; Wherein, the first sample event information is acquired in a third brightness range, the second sample event information is acquired in a fourth brightness range, and the sample scene image is in the fourth brightness range Obtained within, the fourth brightness range is higher than the third brightness range. The device according to claim 13, wherein the image processing network further comprises an identification network, and the training module comprises: The first extraction sub-module is used to input the first sample event information of the first sample scene and the second sample event information of the second sample scene into the first feature extraction network to obtain the first The characteristics of this event and the characteristics of the second sample; The first authentication sub-module is configured to input the first sample event feature and the second sample event feature into the authentication network respectively to obtain a first authentication result and a second authentication result; The first confrontation training sub-module is used to confront the training of the image processing network according to the first identification result and the second identification result. The device according to claim 14, wherein the training module further comprises: A first reconstruction sub-module, configured to input the second sample event feature into the image reconstruction network to obtain a first reconstructed image of the second sample scene; The first training sub-module is used to train the image processing network according to the first reconstructed image of the second sample scene and the sample scene image. The apparatus according to claim 15, wherein the image processing network further comprises a detail enhancement network, and the training module further comprises: The first enhancement submodule is configured to input the second sample event feature and the third noise information into the detail enhancement network to obtain the fourth sample event feature; The first fusion sub-module is used to fuse the second sample event feature with the fourth sample event feature to obtain a second sample fusion feature; A second reconstruction sub-module, configured to input the second sample fusion feature into the image reconstruction network to obtain a third reconstructed image of the second sample scene; The second training submodule is used to train the image processing network according to the first reconstructed image of the second sample scene, the third reconstructed image, and the sample scene image. The apparatus according to claim 14, wherein the image processing network further comprises a second feature extraction network, and the training module further comprises: The second extraction submodule is configured to input the second sample event information and the second noise information of the second sample scene into the second feature extraction network to obtain the third sample event feature; The second fusion sub-module is used to fuse the event feature of the second sample with the event feature of the third sample to obtain the first sample fusion feature; The second authentication sub-module is used to input the fusion feature of the first sample into the authentication network to obtain a third authentication result; The second confrontation training sub-module is used to confront and train the image processing network according to the first identification result and the third identification result. The device according to claim 17, wherein the training module further comprises: A third reconstruction sub-module, configured to input the first sample fusion feature into the image reconstruction network to obtain a second reconstructed image of the second sample scene; The third training sub-module is used to train the image processing network according to the second reconstructed image of the second sample scene and the sample scene image. The apparatus according to claim 18, wherein the image processing network further comprises a detail enhancement network, and the training module further comprises: The second enhancement submodule is configured to input the first sample fusion feature and the fourth noise information into the detail enhancement network to obtain the fifth sample event feature; The third fusion sub-module is used to fuse the fusion feature of the first sample with the event feature of the fifth sample to obtain a third sample fusion feature; A fourth reconstruction submodule, configured to input the third sample fusion feature into the image reconstruction network to obtain a fourth reconstructed image of the second sample scene; The fourth training sub-module is configured to train the image processing network according to the second reconstructed image of the second sample scene, the fourth reconstructed image, and the sample scene image. The device according to claim 19, wherein the fourth training sub-module is used for: Determine the overall loss of the image processing network according to the second reconstructed image of the second sample scene, the fourth reconstructed image, and the sample scene image; Determine the gradient information of the image processing network according to the overall loss; Adjusting the network parameters of the first feature extraction network, the second feature extraction network, the detail enhancement network, and the image reconstruction network according to the gradient information, Wherein, the gradient information of the detail enhancement network is not transmitted to the second feature extraction network. An electronic device including: processor; A memory for storing processor executable instructions; Wherein, the processor is configured to call instructions stored in the memory to execute the method according to any one of claims 1 to 10. A computer-readable storage medium having computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the method according to any one of claims 1 to 10 is realized. A computer program comprising computer readable code, when the computer readable code runs in an electronic device, a processor in the electronic device executes the method for implementing any one of claims 1 to 10 .

Images