CN116168108A - Method and device for generating image from text, storage medium and electronic device - Google Patents

Abstract

The present disclosure relates to the field of image data processing or generating technologies, and in particular, to a method and apparatus for generating an image by using text, a computer readable storage medium, and an electronic device, where the method includes: coding natural language text describing the image to obtain text coding data; fusing the text coding data with first Gaussian noise of a first preset time step to obtain intermediate text data; sequentially processing the intermediate text data corresponding to the time step by using the coded image generation models with different time steps to obtain image coded data; decoding the image coding data to obtain a target image; wherein the set of time steps of the plurality of encoded image generation models completely covers the first preset time step. The technical scheme of the embodiment of the disclosure improves the precision of the image generated by the text and the matching degree with the text semantic information.

Description

Method and device for generating image from text, storage medium and electronic device

technical field

The present disclosure relates to the technical field of image data processing or generation, and in particular, to a method and device for generating an image from text, a computer-readable storage medium, and electronic equipment.

Background technique

Text-based image generation technology has broad application prospects in many scenarios, including the creation of personalized wallpapers for mobile theme providers, acquisition of creative image materials for slideshows, content creation in virtual spaces, and multi-modal dialogue interaction systems.

However, the method for generating an image from text in the related art has low precision, and the degree of matching between the generated image and the semantic information of the text is low.

It should be noted that the information disclosed in the above background section is only for enhancing the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to those of ordinary skill in the art.

Contents of the invention

The purpose of the present disclosure is to provide a method for generating an image from text, a device for generating an image from text, a computer-readable medium, and an electronic device, thereby improving the accuracy of the image generated by text and the matching with text semantic information at least to a certain extent Spend.

According to a first aspect of the present disclosure, there is provided a method for generating an image from text, including: encoding a natural language text describing an image to obtain text encoding data; combining the text encoding data with a first preset time step Gaussian noise is fused to obtain intermediate text data; the intermediate text data corresponding to the time step is sequentially processed by encoding image generation models with different time steps to obtain image encoding data; the image encoding data is decoded to obtain the target image; wherein the set of multiple time steps of the coded image generation model completely covers the first preset time step.

According to the second aspect of the present disclosure, there is provided an apparatus for generating an image from text, including: an encoding module, configured to encode a natural language text describing an image to obtain encoded text data; a fusion module, configured to combine the encoded text data with the The first Gaussian noise of the first preset time step is fused to obtain the intermediate text data; the processing module is used to sequentially process the intermediate text data corresponding to the time step by using encoding image generation models of different time steps Obtaining coded image data; a decoding module; configured to decode the coded image data to obtain a target image; wherein, a set of time steps of a plurality of coded image generation models completely covers the first preset time step.

According to a third aspect of the present disclosure, there is provided a computer-readable medium on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented.

According to a fourth aspect of the present disclosure, there is provided an electronic device, which is characterized in that it includes: one or more processors; and a memory for storing one or more programs, when one or more programs are used by one or more When the processor is executed, one or more processors are made to implement the above method.

An embodiment of the present disclosure provides a method for generating an image from text, which encodes the natural language text describing the image to obtain text coded data; fuses the text coded data with the first Gaussian noise of the first preset time step to obtain Intermediate text data; use coded image generation models with different time steps to sequentially process the intermediate text data corresponding to the time steps to obtain image coded data; decode the image coded data to obtain the target image. Compared with the prior art, the coded image generation model with different time steps is used to sequentially process the intermediate text data corresponding to the time steps to obtain image coded data, which increases the capacity of the model and improves the quality of image generation.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Description of drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description serve to explain the principles of the disclosure. Apparently, the drawings in the following description are only some embodiments of the present disclosure, and those skilled in the art can obtain other drawings according to these drawings without creative efforts. In the attached picture:

FIG. 1 shows a schematic diagram of an exemplary system architecture to which embodiments of the present disclosure can be applied;

FIG. 2 schematically shows a flowchart of a method for generating an image from text in an exemplary embodiment of the present disclosure;

Fig. 3 schematically shows a structural diagram of acquiring natural language text in an exemplary embodiment of the present disclosure;

Fig. 4 schematically shows a flow chart of acquiring image encoding data in an exemplary embodiment of the present disclosure;

Fig. 5 schematically shows a flow chart of acquiring a coded image generation model in an exemplary embodiment of the present disclosure;

FIG. 6 schematically shows a data flow diagram for training a coded image generation model in an exemplary embodiment of the present disclosure;

Fig. 7 schematically shows a flow chart of acquiring a target image in an exemplary embodiment of the present disclosure;

Fig. 8 schematically shows a flowchart of acquiring an intermediate image in an exemplary embodiment of the present disclosure;

Fig. 9 schematically shows a data flow diagram of a training image decoding model in an exemplary embodiment of the present disclosure;

FIG. 10 schematically shows another flowchart for acquiring an intermediate image in an exemplary embodiment of the present disclosure;

FIG. 11 schematically shows another flowchart of acquiring a target image in an exemplary embodiment of the present disclosure;

Fig. 12 schematically shows another flowchart of acquiring a target image in an exemplary embodiment of the present disclosure;

FIG. 13 schematically shows a data flow diagram of a method for generating an image from text in an exemplary embodiment of the present disclosure;

Fig. 14 schematically shows a data flow diagram of a text generated image on a web page in an exemplary embodiment of the present disclosure;

Fig. 15 schematically shows an effect display diagram of generating a preview image on a web page in an exemplary embodiment of the present disclosure;

FIG. 16 schematically shows a composition diagram of an apparatus for generating an image from text in an exemplary embodiment of the present disclosure;

Fig. 17 shows a schematic diagram of an electronic device to which the embodiments of the present disclosure can be applied.

Detailed ways

Example embodiments will now be described more fully with reference to the accompanying drawings. Example embodiments may, however, be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus repeated descriptions thereof will be omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, or in one or more hardware modules or integrated circuits, or in different network and/or processor means and/or microcontroller means.

In related technologies, text-based image generation technology has broad application prospects in many scenarios, including the creation of personalized wallpapers for mobile phone theme providers, the acquisition of PPT creative image materials, content creation in virtual spaces, and multi-modal dialogue interactions. system etc. The main challenge of such open-domain text-generated images is to generate high-quality images containing textual semantic information based on text-based guidance. The current technical solutions are mainly based on GAN model, autoregressive model and diffusion model.

GAN assumes that Z obeys some common distributions (such as normal distribution or uniform distribution), and then hopes to train a model X=g(Z), and learn a discriminator to make the overall judgment of real pictures and generated pictures tend to be consistent . The probability density function is not explicitly modeled, but the generator and the discriminator are confronted. The main disadvantages are unstable training and poor diversity.

The autoregressive model is represented as a Bayesian network by factorization of the chain rule. In the autoregressive generation model, the current node learning depends on the previous generation results, which was first applied in the NLP field of text generation, speech generation, etc., but the image itself is a discrete distribution, composed of a limited number of pixels, and each pixel takes The values are also discrete and finite, and thus can be described by a discrete distribution. The problem with autoregressive is that the generation speed is too slow and the decoding space is too large to generate high-resolution images.

Based on the above shortcomings, the present disclosure provides a method for generating an image from text. FIG. 1 shows a schematic diagram of a system architecture that can implement the above method for generating an image from text. The system architecture 100 may include a terminal 110 and a server 120 . Wherein, the terminal 110 may be a terminal device such as a smart phone, a tablet computer, a desktop computer, or a notebook computer, and the server 120 generally refers to a background system that provides services related to text generation and images in this exemplary embodiment, and may be one server or multiple servers formed clusters. The terminal 110 and the server 120 may be connected through a wired or wireless communication link for data exchange.

In an implementation manner, the above-mentioned method for generating an image from text may be executed by the terminal 110 . For example, the user uses the terminal 110 to obtain a natural language text, and the terminal 110 generates a target image based on the natural language text, and outputs the target image.

In an implementation manner, the server 120 may execute the above-mentioned method for generating an image from text. For example, after a user obtains a natural language text using the terminal 110, the terminal 110 uploads the natural language text to the server 120, and the server 120 generates a target image based on the natural language text, and returns the target image to the terminal 110.

It can be known from the above that the execution subject of the method for generating an image from text in this exemplary embodiment may be the terminal 110 or the server 120, which is not limited in the present disclosure.

The method for generating an image from text in this exemplary embodiment will be described below in conjunction with FIG. 2. FIG. 2 shows an exemplary flow of the method for generating an image from text, which may include:

Step S210, encoding the natural language text describing the image to obtain text encoding data;

Step S220, fusing the text encoding data with the first Gaussian noise of the first preset time step to obtain intermediate text data;

Step S230, using coded image generation models with different time steps to sequentially process the intermediate text data corresponding to the time steps to obtain image coded data;

Step S240, decoding the encoded image data to obtain a target image;

Wherein, the set of multiple time steps of the coded image generation model completely covers the first preset time step.

Based on the above method, the encoding image generation model with different time steps is used to sequentially process the intermediate text data corresponding to the time steps to obtain the image encoding data, which increases the capacity of the model and improves the quality of image generation.

Each step in Fig. 2 is described in detail below.

Referring to FIG. 2, in step S210, the natural language text describing the image is encoded to obtain text encoding data.

In an exemplary embodiment of the present disclosure, the above-mentioned method for generating an image from a text can be applied to a web page, and before encoding the natural language text describing the image to obtain the text encoding data, the natural language describing the image can be obtained first text.

In an example implementation, the natural language base for describing an image may include a picture body, detailed words, and modifiers. Detailed words can be combined arbitrarily, and modifiers can define one style or multiple styles. The basic principle to follow is to conform to the normal Chinese grammatical logic, as shown in Figure 3, for example, "a very detailed digital painting , depicting a person standing in front of a portal, in a mysterious forest of many fantasy trees", and some text descriptions are built into the experience. Then there are style options in the function area, including cartoon, animation, Chinese style, etc., which can also be customized according to user needs. There are personalized label options, including some detailed words and modifiers. Finally, the modifiable settings include image resolution, number of generated images, degree of detail of images, and steps of image generation, etc. The specific details of the natural language text describing images can also be customized according to user needs, which is not done in this example implementation Specific limits.

In this example embodiment, the natural language that describes the image can be obtained through user input and selection. Specifically, as shown in FIG. etc., among them, the style and personalization tags can be optional.

In an example implementation of the present disclosure, after obtaining the above natural language text describing the image, the above natural language text can be encoded to obtain text encoding data, specifically, XLM-Roberta-Large-Vit-B- The 16Plus encoder encodes the natural language text, and other encoders can also be used for encoding, which is not specifically limited in the real-time mode of this example.

In step S220, the text encoding data is fused with the first Gaussian noise of the first preset time step to obtain intermediate text data.

In an exemplary embodiment of the present disclosure, after obtaining the above-mentioned text encoding data, it can be fused with the first Gaussian noise, and the fusion process can include a preset time step, wherein the first preset time step can include 1000 steps, may also be 2000 steps, and may also be customized according to user requirements, which is not specifically limited in this example implementation manner. After fusion, the above-mentioned intermediate text data is obtained.

In step S230, the intermediate text data corresponding to the time steps are sequentially processed by using coded image generation models of different time steps to obtain image coded data.

In the real-time mode of this example, as shown in FIG. 4 , the above steps may include step S410 to step S430.

In step S410, the intermediate text data of the Nth stage is processed by using the coded image generation model corresponding to the Nth stage to obtain the image coded data of the Nth stage.

In this example embodiment, as shown in FIG. 5 , the above method may further include acquiring the coded image generation model, specifically, it may include steps S510 to S530.

In step S510, a first initial model and first training data are acquired, wherein the first training data includes a plurality of reference text coded data and real image coded data corresponding to the reference text coded data.

In an example implementation of the present disclosure, a first initial model and training data may be acquired first, wherein the training data may include multiple reference text encoding data and real image encoding data corresponding to the reference text encoding data.

Wherein, the above-mentioned reference text encoding data is obtained by referring to natural language text encoding, wherein, the above-mentioned reference natural language text encoding may include the open source laion Chinese part in the industry, Wukong data, translated laion Chinese data, collected and crawled data, etc. It can be customized according to user requirements, and is not specifically limited in this example implementation.

Before encoding the above-mentioned reference natural language text data, data cleaning can also be performed on the above-mentioned reference natural language text encoding data. Specifically, filter out descriptions that do not contain physical entity words, and at the same time clean the text content, including removing special punctuation marks, Conversion from complex to simplified, etc., are not specifically limited in this example implementation.

The above-mentioned real-valued image coding data can be obtained after coding the real-valued image, wherein, before encoding the above-mentioned real-valued image, images with low resolution and low aesthetic rating in the real-valued image can be deleted to increase The quality of the training data.

In step S520, input the reference text coding data into the first initial model to obtain reference image coding data.

In an example implementation of the present disclosure, after obtaining the above-mentioned reference text encoding data, the above-mentioned reference text encoding data can be input into the above-mentioned first initial model, wherein the above-mentioned first initial model can be generated by a Transformer Encoder (encoder) Partially composed, random Gaussian noise and the encoding of the time step of the reference text encoding data can be used as conditions to control the generation of the reference image encoding data.

In step S530, the first initial model is trained based on the coded reference image data and the coded image real value data to obtain the coded image generation model.

In an exemplary embodiment of the present disclosure, after obtaining the coded reference image data and the coded real image data, the first initial model may be trained based on the coded reference image data and the coded real image data to obtain the coded image Generate a model.

Specifically, as shown in Figure 6, input the reference natural language text for data encoding to obtain the reference text encoding data, input the reference text encoding data into the above-mentioned first initial model to obtain the reference image encoding data, and use the real value image to encode to obtain the true Value image coding data, using the above-mentioned real value image coding data and the above-mentioned reference image coding data to update the first initial model, so as to obtain a coded image generation model.

In this example embodiment, the above-mentioned first preset time step may include N stages. At this time, the above-mentioned coded image generation models may include N serially arranged coded image generation models, and each coded image generation model processes data of one stage, which can enhance The pertinence of the model improves the quality of the obtained image.

In this example embodiment, the intermediate text data is obtained after the above-mentioned reference text coded image is fused with the first Gaussian noise of the first preset time step, and then the coded image generation model corresponding to the Nth stage is used to generate the Nth The intermediate text data of the first stage is processed to obtain the image coding data of the Nth stage.

In step S420, the coded image data of the Nth stage is processed by using the coded image generation model corresponding to the N-1th stage to obtain the image coded data of the N-1th stage.

In step S430, output the encoded image data of the first stage.

In this exemplary embodiment, the coded image data of the Nth stage is processed by using the coded image generation model corresponding to the N-1th stage to obtain the image coded data of the N-1th stage, that is, the image coded data of the N-1th stage The input to the coded image generative model is the output of the coded image generative model at stage N. The image coding data of the first stage above can be output. It should be noted that the above N is a positive integer greater than 1.

It should be noted that the set of the above-mentioned multiple time steps of the coded image generation model completely covers the first preset time step. For example, if the above-mentioned first preset time step is 1000 steps, then The time step of the above coded image generation model is greater than or equal to 1000 steps, and the step size of each stage in the above N stages can be the same or different, and can be customized according to user needs, and is not specifically limited in this example embodiment .

In step S240, the encoded image data is decoded to obtain a target image.

In an example implementation of the present disclosure, as shown in FIG. 7 , the above steps may include step S710 to step S720.

In step S710, the encoded image data is decoded using an image decoding model to obtain an intermediate image.

In this example implementation, as shown in FIG. 8 , the above steps may include step S820 to step S820.

In step S810, the coded image data is fused with the second Gaussian noise of the second preset time step to obtain an image to be decoded.

In an example implementation of the present disclosure, after obtaining the above image coding data, it can be fused with the second Gaussian noise, and the fusion process can include a preset time step, wherein the second preset time step can include 1000 steps, may also be 2000 steps, and may also be customized according to user requirements, which is not specifically limited in this example implementation manner. Fusion to obtain the above image to be decoded.

In step S820, the images to be decoded corresponding to the time steps are sequentially decoded using image decoding models of different time steps to obtain an intermediate image.

In this example embodiment, as shown in FIG. 9 , the second initial model and the second training data can be obtained first, wherein the second training data includes a plurality of reference images to be decoded and the true intermediate values corresponding to the reference images to be decoded image. Then input the reference image to be decoded into the second initial model to obtain a reference intermediate image, and finally train the second initial model based on the reference intermediate image and the true intermediate image to update the second initial model to obtain an image decoding model.

For the specific training process, reference may be made to the training of the above coded image generation model, which will not be repeated in this example implementation.

In the real-time mode of this example, the input of the above image decoding model can include random third Gaussian noise, image encoding data and training time steps as conditions, and further, it can also be used by randomly zeroing the text encoding data for a certain proportion of time Diffusion model to do the policy of the conditional generation task to generate the above intermediate image.

In this example embodiment, the second preset time step includes M stages, as shown in FIG. Decoding to obtain an intermediate image may include steps S1010 to S1030.

In step S1010, the image to be decoded at the Mth stage is decoded using the image decoding model corresponding to the Mth stage to obtain an intermediate image at the Mth stage.

In this exemplary embodiment, the second preset time step may include M stages, and at this time, the above may include M image decoding models set in series, and each image decoding model processes data of one stage, which can enhance the model's Pertinence, improve the quality of the obtained image.

In this example embodiment, the image to be decoded at the Mth stage is processed by using the image decoding model corresponding to the Mth stage to obtain an intermediate image at the Mth stage.

In step S1020, the intermediate image of the M-th stage is decoded by using the image decoding model corresponding to the M-1-th stage to obtain the intermediate image of the M-1-th stage.

In step S1030, the intermediate image of the first stage is output.

In this example embodiment, the intermediate image of the Mth stage is processed by using the image decoding model corresponding to the M-1th stage to obtain the intermediate image of the M-1th stage, that is, the image decoding model of the M-1th stage The input to is the output of the M-th stage image decoding model. The intermediate image of the first stage above can be output. It should be noted that the above M is a positive integer greater than 1.

It should be noted that the set of the above-mentioned multiple time steps of the image decoding models completely covers the second preset time step. For example, if the above-mentioned second preset time step is 1000 steps, then the above-mentioned The time step of the image decoding model is greater than or equal to 1000 steps, and the step size of each of the above M stages can be the same or different, and can be customized according to user needs, and is not specifically limited in this example embodiment.

In step S720, at least one super-resolution process is performed on the intermediate image to obtain the target image.

In an example implementation of the present disclosure, as shown in FIG. 11 , the above steps may include step S1110 to step S1120 .

In step S1110, the intermediate image is fused with the third Gaussian noise of the third preset time step to obtain an intermediate super-resolution image.

In an example implementation of the present disclosure, after obtaining the above-mentioned intermediate image, it can be fused with the third Gaussian noise, and the fusion process can include a preset time step, wherein the third preset time step can include 1000 steps , can also be 2000 steps, and can also be customized according to user requirements, which is not specifically limited in this example implementation. Fusion to obtain the above image to be decoded.

In step S1120, the intermediate super-resolution images corresponding to the time steps are sequentially super-resolved using image super-resolution models of different time steps to obtain a target image.

In this example embodiment, the third initial model and the third training data may be obtained first, wherein the third training data includes a plurality of reference intermediate images and the ground-truth super-resolution images corresponding to the reference intermediate images. Then input the reference intermediate image into the third initial model to obtain a reference super-resolution image, and finally train the third initial model based on the reference super-resolution image and the true value super-resolution image to update the above-mentioned third initial model to obtain an image super-resolution model.

For the specific training process, reference may be made to the training of the above coded image generation model, which will not be repeated in this example implementation.

In an example implementation of the present disclosure, as shown in FIG. 12 , the third preset time step includes P stages; image super-resolution models with different time steps are used to sequentially perform all stages corresponding to the time steps Performing super-resolution on the intermediate super-resolution image to obtain the target image may include steps S1210 to S1230.

In step S1210, super-resolution is performed on the intermediate image of the Pth stage by using the image super-resolution model corresponding to the Pth stage to obtain the target image of the Pth stage.

In this example embodiment, the third preset time step may include P stages, and at this time, the above may include P image super-resolution models arranged in series, and each image super-resolution model processes data of one stage, It can enhance the pertinence of the model and improve the quality of the obtained image.

In this example embodiment, the intermediate text data is obtained after the above-mentioned intermediate image is fused with the third Gaussian noise of the above-mentioned third preset time step, and then the image super-resolution model corresponding to the P-th stage is used to process the P-th The intermediate image of the stage is processed to obtain the target image of the Pth stage.

In step S1220, super-resolution is performed on the intermediate image of the Pth stage by using the image super-resolution model corresponding to the P-1th stage to obtain the target image of the P-1th stage.

In step S1230, the target image of the first stage is output.

In this example embodiment, the target image of the P-th stage is processed by using the image super-resolution model corresponding to the P-1th stage to obtain the target image of the P-1th stage, that is, the image of the P-1th stage The input to the super-resolution model is the output of the P-stage image super-resolution model. The target image of the first stage above can be output. It should be noted that the above P is a positive integer greater than 1.

It should be noted that the above-mentioned set of multiple time steps of the image super-resolution model completely covers the third preset time step. For example, if the above-mentioned third preset time step is 1000 steps, Then the time step size of the above-mentioned image super-resolution model is greater than or equal to 1000 steps, and the step size of each stage in the above-mentioned P stages can be the same or different, and can be customized according to user needs, which is not done in this example embodiment Specific limits.

In this example embodiment, the number of the above-mentioned image super-resolution models may be one or multiple. In an example embodiment, after using the above-mentioned image super-resolution model to obtain the target image, SwinIR may also be used The super-resolution model updates the above target image.

The method for generating an image from the above-mentioned text is described in detail below with reference to FIG. 13. First, the above-mentioned natural language text can be input into the above-mentioned text encoding module to obtain text encoding data. The encoded data is input to the image decoding model to obtain an intermediate image, and then the above-mentioned image super-resolution model is used to perform super-resolution on the above-mentioned intermediate image to obtain the target image, wherein, the SwinIR super-resolution model can also be used to perform super-resolution on the above-mentioned target image to update The above target image.

In this example embodiment, the resolution of the above-mentioned intermediate image may be 64*64, the resolution of the target image after the image super-resolution model may be 256*256, and the above-mentioned target image is super-resolved through the SwinIR super-resolution model The resolution of the target image above may be 1024*1024 after the update rate. The specific numerical values of the foregoing resolutions can be customized according to user requirements, and are not specifically limited in this exemplary embodiment.

In an example implementation of the present disclosure, the above method can be applied to the web page. Specifically, as shown in FIG. The text is sent to the text graph model, and the text graph model can generate a target image by using the above-mentioned method for generating an image from text, and feed it back to the user via the web server.

Referring to Figure 15, the user can preview the image on the web page. After the user clicks the generated image logo, the generated image will be displayed on the image display interface. The number of generated images can be customized according to the user's needs. In this example implementation is not specifically limited.

To sum up, in this exemplary embodiment, the coded image generation models with different time steps are used to sequentially process the intermediate text data corresponding to the time steps to obtain image coded data, which increases the capacity of the model and improves the quality of image generation. At the same time, the image decoding model with different time steps is used to encode and process the image encoding data corresponding to the time step at one time, and the image super-resolution model that does not pass the time step is used to super-resolution the intermediate image corresponding to the time step. Further improve the quality of generated images. Among them, the image super-resolution model, the coded image generation model and the image decoding model all use the diffusion model. The text generated image has a strong fitting ability for complex distribution and big data, and at the same time, the generation quality is high, the diversity is good, and it can be edited. strong ability.

It should be noted that the above-mentioned figures are only schematic illustrations of processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not imply or limit the chronological order of these processes. In addition, it is also easy to understand that these processes may be executed synchronously or asynchronously in multiple modules, for example.

Further, referring to FIG. 16 , the embodiment of this example also provides an apparatus 1600 for generating images from text, including an encoding module 1610 , a fusion module 1620 , a processing module 1630 and a decoding module 1640 . in:

The encoding module 1610 may be used to encode the natural language text describing the image to obtain text encoding data.

The fusion module 1620 may be configured to fuse the encoded text data with the first Gaussian noise of the first preset time step to obtain intermediate text data.

The processing module 1630 may be configured to sequentially process the intermediate text data corresponding to the time steps using coded image generation models of different time steps to obtain image coded data.

In an example implementation, the processing module 1630 may be configured to acquire a first initial model and first training data, wherein the first training data includes a plurality of reference text encoding data and corresponding Real-value image coding data; input the reference text coding data into the first initial model to obtain reference image coding data; train the first initial model based on the reference image coding data and the real-value image coding data to obtain The coded image generates a model.

In an example implementation, the first preset time step includes N stages, and the processing module 1630 may be configured to use the coded image generation model corresponding to the Nth stage to process the intermediate text data of the Nth stage Process the encoded image data at the Nth stage; use the encoded image generation model corresponding to the N-1st stage to process the encoded image data at the Nth stage to obtain the encoded image data at the N-1 stage; The image coding data output of the first stage; wherein, N is a positive integer greater than 1.

The decoding module 1640 may be configured to decode the encoded image data to obtain a target image.

In an example implementation, the decoding module 1640 may be configured to use an image decoding model to decode the encoded image data to obtain an intermediate image; perform super-resolution processing on the intermediate image at least once to obtain the target image.

In an example implementation, the decoding module 1640 may be configured to fuse the encoded image data with the second Gaussian noise of the second preset time step to obtain an image to be decoded; using image decoding models with different time steps Decoding the image to be decoded corresponding to the time step in sequence to obtain an intermediate image; wherein, a set of time steps of multiple image decoding models completely covers the second preset time step.

In this example embodiment, the decoding module 1640 may be configured to use the image decoding model corresponding to the Mth stage to decode the image to be decoded at the Mth stage to obtain an intermediate image at the Mth stage; The image decoding model corresponding to the stage decodes the intermediate image of the Mth stage to obtain the intermediate image of the M-1th stage; outputs the intermediate image of the first stage; wherein, M is a positive integer greater than 1.

In an example implementation, the decoding module 1640 may be configured to fuse the intermediate image with the third Gaussian noise of the third preset time step to obtain an intermediate super-resolution image; image super-resolution using different time steps The rate model sequentially performs super-resolution on the intermediate super-resolution image corresponding to the time step to obtain the target image; wherein, the set of time steps of multiple image super-resolution models completely covers the third preset Time Step.

In this example embodiment, the third preset time step includes P stages, and the decoding module 1640 may be configured to use the image super-resolution model corresponding to the P-th stage to perform super-resolution on the intermediate image of the P-th stage resolution to obtain the target image of the Pth stage; use the image super-resolution model corresponding to the P-1th stage to perform super-resolution on the intermediate image of the P-th stage to obtain the target image of the P-1th stage; The target image output of the first stage; wherein, P is a positive integer greater than 1.

The specific details of each module in the above device have been described in detail in the implementation of the method, and details not disclosed can be found in the implementation of the method, so details are not repeated here.

Exemplary embodiments of the present disclosure also provide an electronic device for performing the above-mentioned method for generating an image from text, and the electronic device may be the above-mentioned terminal 110 or server 120 . Generally, the electronic device may include a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to execute the above-mentioned method for generating an image from text by executing the executable instructions.

The following takes the mobile terminal 1700 in FIG. 17 as an example to illustrate the structure of the electronic device. Those skilled in the art will appreciate that, in addition to components specifically intended for mobile purposes, the configuration in Fig. 17 can also be applied to equipment of a stationary type.

As shown in Figure 17, the mobile terminal 1700 may specifically include: a processor 1701, a memory 1702, a bus 1703, a mobile communication module 1704, an antenna 1, a wireless communication module 1705, an antenna 2, a display screen 1706, a camera module 1707, and an audio module 1708 , a power module 1709 and a sensor module 1710 .

The processor 1701 may include one or more processing units, for example: the processor 1701 may include an AP (Application Processor, application processor), a modem processor, a GPU (Graphics Processing Unit, a graphics processor), an ISP (Image Signal Processor , image signal processor), controller, encoder, decoder, DSP (Digital Signal Processor, digital signal processor), baseband processor and/or NPU (Neural-Network Processing Unit, neural network processor), etc. The method for generating an image from text in this exemplary embodiment may be performed by an AP, GPU or DSP, and when the method involves neural network-related processing, it may be performed by an NPU.

An encoder can encode (i.e. compress) an image or video, for example, it can encode a target image into a specific format to reduce the data size for easy storage or transmission. The decoder can decode (i.e. decompress) the encoded data of the image or video to restore the image or video data. For example, the encoded data of the target image can be read and decoded by the decoder to restore the data of the target image. Further, the relevant processing of the text to generate the image is performed on the data. The mobile terminal 200 may support one or more encoders and decoders. In this way, the mobile terminal 200 can process images or videos in multiple encoding formats, such as: JPEG (Joint Photographic Experts Group, Joint Photographic Experts Group), PNG (Portable Network Graphics, portable network graphics), BMP (Bitmap, bitmap) etc. Image format, MPEG (Moving Picture Experts Group, moving picture expert group) 1, MPEG2, H.263, H.264, HEVC (High Efficiency Video Coding, high-efficiency video coding) and other video formats.

The processor 1701 can form a connection with the memory 1702 or other components through the bus 1703 .

Memory 1702 may be used to store computer-executable program code, which includes instructions. The processor 1701 executes various functional applications and data processing of the mobile terminal 1700 by executing instructions stored in the memory 1702 . The storage 1702 can also store application data, such as storing images, videos and other files.

The communication function of the mobile terminal 1700 can be realized by the mobile communication module 1704, the antenna 1, the wireless communication module 1705, the antenna 2, the modem processor and the baseband processor, etc. Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 1704 can provide 2G, 3G, 4G, 5G and other mobile communication solutions applied on the mobile terminal 1700 . The wireless communication module 1705 can provide wireless communication solutions such as wireless local area network, bluetooth, and near field communication applied on the mobile terminal 1700 .

The display screen 1706 is used to implement display functions, such as displaying user interfaces, images, videos, and the like. The camera module 1707 is used to realize the shooting function, such as taking images and videos. The audio module 208 is used to implement audio functions, such as playing audio, collecting voice, and so on. The power module 209 is used to implement power management functions, such as charging the battery, supplying power to the device, and monitoring the state of the battery. The sensor module 1710 may include a depth sensor 17101, a pressure sensor 17102, a gyroscope sensor 17103, an air pressure sensor 17104, etc., so as to realize corresponding sensing and detection functions.

Those skilled in the art can understand that various aspects of the present disclosure can be implemented as a system, method or program product. Therefore, various aspects of the present disclosure can be embodied in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which can be collectively referred to herein as "circuit", "module" or "system".

Exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above-mentioned method in this specification is stored. In some possible implementations, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program code. When the program product runs on the terminal device, the program code is used to make the terminal device execute the above-mentioned Steps according to various exemplary embodiments of the present disclosure described in the "Exemplary Methods" section.

It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. A computer readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable Programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination of the above.

In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave carrying computer-readable program code therein. Such propagated data signals may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. . Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

Additionally, program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages—such as Java, C++, etc., as well as conventional procedural Programming language - such as "C" or a similar programming language. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server to execute. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (for example, using an Internet service provider). business to connect via the Internet).

Other embodiments of the disclosure will be readily apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any modification, use or adaptation of the present disclosure, and these modifications, uses or adaptations follow the general principles of the present disclosure and include common knowledge or conventional technical means in the technical field not disclosed in the present disclosure . The specification and examples are to be considered exemplary only, with the true scope and spirit of the disclosure indicated by the appended claims.

It should be understood that the present disclosure is not limited to the precise constructions which have been described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims (11)

1. A method for generating images from text, comprising: Encoding the natural language text describing the image to obtain text encoding data; Fusing the text encoding data with the first Gaussian noise of the first preset time step to obtain intermediate text data; Using coded image generation models with different time steps to sequentially process the intermediate text data corresponding to the time steps to obtain image coded data; Decoding the encoded image data to obtain a target image; Wherein, the set of multiple time steps of the coded image generation model completely covers the first preset time step. 2. The method according to claim 1, characterized in that the method further comprises: Acquiring a first initial model and first training data, wherein the first training data includes a plurality of reference text encoding data and real-value image encoding data corresponding to the reference text encoding data; inputting the reference text coded data into the first initial model to obtain reference image coded data; training the first initial model based on the reference image encoding data and the true image encoding data to obtain the encoded image generation model. 3. The method according to claim 1, wherein the first preset time step size comprises N stages; the coded image generation model utilizing different time step sizes sequentially applies the corresponding time step size The intermediate text data is processed to obtain image coding data, including: Processing the intermediate text data of the Nth stage by using the coded image generation model corresponding to the Nth stage to obtain the image coded data of the Nth stage; Processing the coded image data at the Nth stage by using the coded image generation model corresponding to the N-1 stage to obtain the coded image data at the N-1 stage; output the image coding data of the first stage; Wherein, N is a positive integer greater than 1. 4. The method according to claim 1, wherein said decoding said encoded image data to obtain a target image comprises: Using an image decoding model to decode the encoded image data to obtain an intermediate image; performing at least one super-resolution process on the intermediate image to obtain the target image. 5. The method according to claim 4, wherein the decoding of the encoded image data using an image decoding model to obtain an intermediate image comprises: Fusing the encoded image data with second Gaussian noise at a second preset time step to obtain an image to be decoded; Using image decoding models with different time steps to sequentially decode the images to be decoded corresponding to the time steps to obtain an intermediate image; Wherein, a set of multiple time steps of the image decoding models completely covers the second preset time step. 6. The method according to claim 5, wherein the second preset time step comprises M stages; the image decoding models corresponding to the time steps are sequentially performed on the waiting time steps corresponding to the time steps. The decoded image is processed to obtain an intermediate image, including: Using the image decoding model corresponding to the Mth stage to decode the image to be decoded at the Mth stage to obtain an intermediate image at the Mth stage; Using the image decoding model corresponding to the M-1th stage to decode the intermediate image of the M-th stage to obtain the intermediate image of the M-1th stage; outputting the intermediate image of the first stage; Wherein, M is a positive integer greater than 1. 7. The method according to claim 4, wherein said performing at least one super-resolution process on said intermediate image to obtain said target image comprises: Fusing the intermediate image with the third Gaussian noise of the third preset time step to obtain an intermediate super-resolution image; Using image super-resolution models with different time steps to sequentially perform super-resolution on the intermediate super-resolution images corresponding to the time steps to obtain a target image; Wherein, the set of multiple time steps of the image super-resolution model completely covers the third preset time step. 8. The method according to claim 7, wherein the third preset time step comprises P stages; image super-resolution models of different time steps are used to sequentially process all the time steps corresponding to the time step Perform super-resolution on the intermediate super-resolution image to obtain the target image, including: Using the image super-resolution model corresponding to the P-th stage to perform super-resolution on the intermediate image of the P-th stage to obtain the target image of the P-th stage; Using the image super-resolution model corresponding to the P-1th stage to perform super-resolution on the intermediate image of the P-th stage to obtain the target image of the P-1th stage; output the target image of the first stage; Wherein, P is a positive integer greater than 1. 9. A device for generating images from text, comprising: The encoding module is used to encode the natural language text describing the image to obtain text encoding data; A fusion module, configured to fuse the encoded text data with the first Gaussian noise of the first preset time step to obtain intermediate text data; A processing module, configured to sequentially process the intermediate text data corresponding to the time steps using coded image generation models of different time steps to obtain coded image data; A decoding module; used to decode the encoded image data to obtain a target image; Wherein, the set of multiple time steps of the coded image generation model completely covers the first preset time step. 10. A computer-readable storage medium, on which a computer program is stored, wherein when the program is executed by a processor, the method for generating an image from text according to any one of claims 1 to 8 is realized. 11. An electronic device, characterized in that it comprises: one or more processors; and A memory for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement any one of claims 1 to 8 Method for generating images from text as described in item.

Images