CN115170390B

CN115170390B - File stylization method, device, equipment and storage medium

Info

Publication number: CN115170390B
Application number: CN202211061947.4A
Authority: CN
Inventors: 王传鹏; 周惠存; 黄坚林; 孙尔威; 陈春梅; 张键; 林依婷
Original assignee: Guangzhou Jishang Network Technology Co ltd
Current assignee: Guangzhou Jishang Network Technology Co ltd
Priority date: 2022-08-31
Filing date: 2022-08-31
Publication date: 2023-01-06
Anticipated expiration: 2042-08-31
Also published as: CN115170390A

Abstract

The invention discloses a file stylization method, a device, equipment and a storage medium, wherein the method comprises the following steps: loading a generation countermeasure network of a fixed structure; inquiring and generating a plurality of configuration files which are all adaptive to the countermeasure network, wherein each configuration file records parameters which are obtained by training the countermeasure network and take a certain reconstructed image style as a target; loading each configuration file respectively for generating the countermeasure network; receiving a stylization request sent by a first client for calling a stylization service for an original data object; responding to the stylization request, and determining the image style of the original data object to be reconstructed as a target style; searching a generation countermeasure network of the loaded target file as an image style reconstruction network; and inputting the original data object into an image style reconstruction network to be reconstructed into a target data object. When the image style is newly added, the project of the prior image style can be inherited, the repeated development work is reduced, the development time consumption is reduced, and the development timeliness is improved.

Description

File stylization method, device, equipment and storage medium

Technical Field

The invention relates to the technical field of computer vision, in particular to a file stylizing method, device, equipment and storage medium.

Background

In scenes such as photographing, making social messages, recording short videos and the like, a user can make various different image data and video data, and in order to improve the quality of the image data and the video data, the user usually performs post-processing on the video data after acquiring the image data and the video data.

One of the common post-processing is to change the style of the picture, and the deviation of different users to the style is different, except that the traditional styles such as animation, oil painting, drawing and the like are relatively stable, some styles become hot spots in a short period along with the rise of different services, for example, animation adapted from a certain popular game shows a theme song with high popularity, so that the style of the animation picture becomes hot spots in a short period, the download quantity of a certain cure system of mobile games is high, so that the style of the mobile game picture becomes hot spots in a short period, and the like.

At present, the neural network is mostly used for stylizing image data and video data, but if each style is added, an independent neural network is trained according to the style of the picture, so that the development amount is high, the time consumption is long, certain hysteresis exists, in addition, the number of the neural networks is continuously accumulated in the process of accumulating different styles, the structure of the neural network is usually huge, the numerous neural networks not only occupy a large amount of storage resources, but also cause great burden on resources such as a processor, a memory and the like during the operation.

Disclosure of Invention

The invention provides a file stylizing method, a device, equipment and a storage medium, aiming at solving the problem of how to improve the timeliness of the style of a newly added picture and reduce the burden on various resources.

According to an aspect of the present invention, there is provided a document stylization method, including:

loading a generation countermeasure network of a fixed structure;

inquiring a plurality of configuration files which are matched with the generated countermeasure network, wherein each configuration file records parameters obtained by training the generated countermeasure network and aiming at reconstructing a certain image style;

loading each configuration file for the generated countermeasure network respectively;

receiving a stylization request sent by a first client for calling a stylization service for an original data object;

in response to the stylization request, determining the image style of the original data object to be reconstructed as a target style;

searching the generated countermeasure network loaded with a target file as an image style reconstruction network, wherein the target file is the configuration file recorded with parameters obtained by training the generated countermeasure network with the target style reconstructed as a target;

and inputting the original data object into the image style reconstruction network to reconstruct the original data object into a target data object, wherein the target data object maintains the content of the original data object and has the target style.

According to another aspect of the present invention, there is provided a document stylizing apparatus comprising:

the generation countermeasure network loading module is used for loading the generation countermeasure network of the fixed structure;

the configuration file query module is used for querying a plurality of configuration files which are matched with the generated confrontation network, and each configuration file records parameters which are obtained by training the generated confrontation network and take a certain image style as a target;

the configuration file loading module is used for respectively loading each configuration file for the generated countermeasure network;

the stylized request receiving module is used for receiving a stylized request sent by a first client for calling a stylized service for an original data object;

the target style determining module is used for responding to the stylization request and determining the image style of the original data object to be reconstructed as a target style;

the image style reconstruction network searching module is used for searching the generated countermeasure network loaded with a target file as an image style reconstruction network, wherein the target file is the configuration file recorded with parameters obtained by training the generated countermeasure network with the reconstructed target style as a target;

and the target data object reconstruction module is used for inputting the original data object into the image style reconstruction network to reconstruct the original data object into a target data object, and the target data object maintains the content of the original data object and has the target style.

According to another aspect of the present invention, there is provided an electronic apparatus including:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

the memory stores a computer program executable by the at least one processor, the computer program being executable by the at least one processor to enable the at least one processor to perform the file stylization method of any one of the embodiments of the present invention.

According to another aspect of the present invention, there is provided a computer-readable storage medium storing a computer program for causing a processor to implement a file stylizing method according to any one of the embodiments of the present invention when executed.

In this embodiment, a generative countermeasure network of a fixed structure is loaded; inquiring and generating a plurality of configuration files which are all adaptive to the antagonistic network, wherein each configuration file records parameters obtained by training the antagonistic network by taking a certain image style as a target; loading each configuration file respectively for generating the countermeasure network; receiving a stylization request sent by a first client for calling a stylization service for an original data object; responding to the stylization request, and determining the image style of the original data object to be reconstructed as a target style; searching a generation countermeasure network of a loaded target file as an image style reconstruction network, wherein the target file is a configuration file recorded with parameters obtained by training the generation countermeasure network by taking a reconstruction target style as a target; the original data object is input into an image style reconstruction network and reconstructed into a target data object, and the target data object maintains the content of the original data object and has a target style. The embodiment provides a generation countermeasure network with a unified structure to realize various image styles, a countermeasure mechanism in the generation countermeasure network has universality, a discriminator is used for learning and judging the image style of image data, the operations of designing an objective function for measuring the stylization quality and the like can be reduced, when an image style is newly added, the project of the prior image style can be inherited, the repeated development work is reduced, the technical threshold of development is greatly reduced, the development workload is reduced, the development time consumption is reduced, the development timeliness is improved, in the process of continuously accumulating the image style, the structure of the generation countermeasure network is fixed into one, configuration files are mainly accumulated, the management is convenient, the expansibility is strong, the stylization can be realized by loading the configuration files into the generation countermeasure network as required, the occupation of storage resources can be greatly reduced, and the burden on resources such as a processor, a memory and the like can be reduced during the operation.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present invention, nor do they necessarily limit the scope of the invention. Other features of the present invention will become apparent from the following description.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.

FIG. 1 is a flowchart of a document stylizing method according to an embodiment of the present invention;

FIG. 2 is a stylized architecture diagram provided in accordance with an embodiment of the present invention;

FIG. 3 is a flowchart of a document stylization method according to a second embodiment of the present invention;

FIG. 4 is a flowchart of a document stylization method according to a third embodiment of the present invention;

fig. 5 is a schematic structural diagram of a document stylizing apparatus according to a fourth embodiment of the present invention;

fig. 6 is a schematic structural diagram of an electronic device for implementing the fifth embodiment of the present invention.

Detailed Description

In order to make those skilled in the art better understand the technical solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in other sequences than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

Example one

Fig. 1 is a flowchart of a file stylization method according to an embodiment of the present invention, where the embodiment is applicable to a case where parameters of different styles are implemented by using a Generative Adaptive Network (GAN) training with a unified structure, so as to perform stylization according to a service loading parameter, the method may be executed by a file stylization apparatus, the file stylization apparatus may be implemented in a hardware and/or software manner, and the file stylization apparatus may be configured in an electronic device. As shown in fig. 1, the method includes:

and step 101, loading a generation countermeasure network of a fixed structure.

In the present embodiment, as shown in fig. 2, a generation countermeasure network 210 may be constructed in advance for different image styles (i.e., styles of pictures), that is, the structure of the generation countermeasure network 210 is fixed.

The generation countermeasure network with the fixed structure is uniformly trained according to various image styles, and multiple parameters suitable for the generation countermeasure network with the fixed structure can be obtained.

Generally, generating a countermeasure network includes a generator and an arbiter. Wherein the generator is responsible for generating content from the random vector, which in this embodiment is image data, in particular image data having the style of an animation adapted by the game; the discriminator is responsible for discriminating whether the received content is authentic, and the discriminator usually gives a probability representing the authenticity of the content.

The generator and the discriminator may use different structures, and for the function of processing the image data, the structures are not limited to the artificially designed Neural network, such as Convolutional Layer (Convolutional Layer), fully Connected Layer (full Connected Layers), etc., the Neural network may also be optimized by a model quantization method, the Neural network for stylized Search by a NAS (Neural network Architecture Search) method, etc., which is not limited in this embodiment.

For generators and discriminators of different structures, the generation countermeasure network can be classified into the following types:

DCGAN (deep convolution generated countermeasure network), CGAN (conditional generated countermeasure network), cycleGAN (periodic generated countermeasure network), coGAN (coupled generated countermeasure network), proGAN (incremental growth of generated countermeasure network), WGAN (Wasserstein generated countermeasure network), SAGAN (self attention generated countermeasure network), bigGAN (large generated countermeasure network), stylegagan (style based generated countermeasure network).

The generator and the discriminator have a countermeasure, so called countermeasure, which can mean in the process of generating an alternative training of the countermeasure network, taking the generation of image data with a designated style as an example, the generator generates some false image data and true image data, the false image data and the true image data are sent to the discriminator to be discriminated together, the discriminator learns and distinguishes the two, the true image data (namely, the image data with the designated style) is given a high score, the false image data (namely, the image data without the designated style) is given a low score, after the discriminator can skillfully judge the existing image data, the generator is targeted to obtain a high score from the discriminator, better false image data is continuously generated until the discriminator can be cheated, the process is repeated until the prediction probability of the discriminator to any image data is close to 0.5, namely, the image data can not be distinguished to be true or false, and the training can be stopped.

Further, the sample for training the generation of the countermeasure network may be selected as paired data (paired data), which may improve performance of generation of the countermeasure network in consideration of the difficulty in collecting the paired data in some cases, for example, if the image style is an animation style, the image style source (sample) for training the generation of the countermeasure network is image data depicting an animation, in order to maintain the paired data, the content source (sample) for training the generation of the countermeasure network is content source (sample) for training the pairing data including image data of the animation corresponding to the real world, and so on, and thus, for some image styles, the sample for training the generation of the countermeasure network is unpaired data (unpaired data) training, that is, the generation of the countermeasure network supports unpaired data training, for example, cycleGAN, style gan, and so on.

The generation of the confrontation network introduces a discriminator, the universality of a confrontation mechanism of the discriminator reduces the difficulty of designing the generation of the confrontation network, and particularly, objective functions which are difficult to directly measure the stylization by using a mathematical formula are used for supporting the reconstruction of various image styles.

For different image styles, if a loss function based on norm needs to be used, a large amount of paired data is needed, the image style of image data is learned and judged by a discriminator in the countermeasure network, and the discriminator learns the image style instead of the object in the image data, so that the image data conforming to the same image style can support training without depending on the object in the corresponding image data.

Taking Learning to cartonize Using White-box cartoonification networks as an example of generating an antagonistic network, which comprises three modules, the original image data and the stylized image data can be divided into three Representations:

1. surface characterization

Surface characterizations are extracted to represent a smooth surface of the image data. Given image data, weighted low frequency components may be extracted, where color components and surface texture are preserved, edges, texture, and details are ignored, and may be used to achieve a flexible and learnable feature representation of a smooth surface.

2. Structure characterization

The structural representation can effectively grasp global structural information and sparse color blocks in the celluloid cartoon style to extract the segmentation areas from the input image data, and an adaptive coloring algorithm is applied to each segmentation area to generate the structural representation. The structural representation can imitate the celluloid cartoon style and is characterized by clear boundary and sparse color blocks.

3. texture characterization

Texture characterization contains the details and edges of the rendering. The input image data is converted to a single channel intensity map with color and brightness removed and the relative pixel intensities preserved. Texture characterization can direct the network to learn high frequency texture details independently, excluding color and brightness patterns.

And when training, controlling the style of image data output by balancing the weight of the surface representation, the structure representation and the texture representation.

The structure of the generated countermeasure network can be read by applying the stylized service, and the generated countermeasure network is loaded to the memory for operation.

And 102, inquiring and generating a plurality of configuration files adaptive to the countermeasure network.

In the present embodiment, as shown in fig. 2, the generative countermeasure network 210 with fixed structure is uniformly trained for multiple image styles offline, so that multiple sets of parameters suitable for the generative countermeasure network with fixed structure can be obtained, each set of parameters can be stored in the configuration file config212 of the generative countermeasure network, and stored in a local database, that is, each configuration file config212 records parameters obtained by training the generative countermeasure network with the goal of reconstructing a certain image style.

When the stylized service is applied, a plurality of configuration files which are matched with the generated countermeasure network are read from a local database.

And 103, respectively loading each configuration file for generating the countermeasure network.

In this embodiment, each configuration file is loaded into the instance of the generative confrontation network, that is, a parameter trained for a certain image style is loaded into the generative confrontation network, so that the generative confrontation network has the capability of reconstructing the image style corresponding to the parameter.

The generative countermeasure network is encapsulated into a main class, the generative countermeasure network generally operates as an instance (i.e., an instantiated object), while in the electronic device, the resources allocated to the generative countermeasure network may vary, depending on the type of electronic device, the priority of the stylized service, etc., and the number of instances created for the generative countermeasure network may also vary.

In the case of sufficient resources, as shown in fig. 2, a plurality of instances 211 may be generated for the countermeasure network 210, where the number of instances 211 is the same as the number of profiles config212, that is, one instance 211 is created for each image style.

Further, a plurality of processes can be created, a process being a dynamic execution of a program on a data set, wherein the number of processes is the same as the number of profiles, i.e. one process is created for each image style, and one instance is created in each process for generating the countermeasure network.

Each profile is thus loaded into each instance separately, building a generative countermeasure network (i.e., instance) for each image style with the ability to reconstruct that image style.

It should be noted that, the configuration files of all image styles are not necessarily loaded simultaneously in the generation of the countermeasure network, the configuration files of corresponding image styles can be loaded according to the requirements of the service, the configuration files of corresponding image styles are released in an idle state, and the occupation of resources is reduced.

In the case of resource shortage, one or more instances can be generated for the generation countermeasure network, wherein the number of the instances is smaller than the number of the configuration files, that is, the partial image styles share one instance, and the configuration files of different image styles are switched and loaded according to a certain scheduling sequence.

Further, one or more processes may be created in which an instance is created for the reactive network, wherein the number of processes is less than the number of profiles.

For the example of the current partial image style sharing, the next image style to be reconstructed can be queried in the scheduled order as the sample style.

And if the sample style is different from the current image style, traversing each configuration file aiming at the sample style.

And if a certain configuration file records a parameter obtained by generating a countermeasure network training by taking the sample style as a reconstruction target, loading the configuration file into the instance.

If the sample style is the same as the current image style, then the current profile is maintained in this example and not replaced.

And 104, receiving a stylization request sent by a first client for calling the stylization service for the original data object.

Generally, the structure and parameters of the generated countermeasure network are huge, and the resources occupied by the generated countermeasure network are large, and the generated countermeasure network is usually deployed in a server, the server can provide a stylized service for a user of a local area network or a public network by using an Application Programming Interface (API), the user can log in a first client, and the first client calls the API to transmit an original data object of an image style to be reconstructed to the server, and sends a stylized request to the server.

For the purpose of format differentiation, the original data object may include image data and video data, and for the convenience of differentiation, the original data object is respectively referred to as original image data and original video data.

In distinction from scenes, the original data objects uploaded by users in the local area network may include image data and video data related to business, such as posters, short videos designed by artists (users) to promote certain games, and the like, and the original data objects uploaded by users in the public network are mostly image data and video data related to life, entertainment and work of users, such as self-photographing of users, short videos recorded by users, and the like.

Step 105, responding to the stylization request, determining the image style of the original data object to be reconstructed as the target style.

The user can select the image style reconstructed from the original data object according to the requirements of business and the like, and the image style is recorded and encapsulated in the stylization request by using the specific zone bit.

And step 106, searching the generation countermeasure network of the loaded target file as an image style reconstruction network.

In this embodiment, an object file may be defined for the target style, and the object file is a configuration file recorded with parameters obtained by training a pair of countermeasure networks for the purpose of reconstructing the target style.

In the generation countermeasure network (namely an example) of the loaded configuration file, the generation countermeasure network of the loaded target file is searched and is marked as an image style reconstruction network.

And step 107, inputting the original data object into an image style reconstruction network to be reconstructed into a target data object.

In this embodiment, the original data object is input into an image style reconstruction network, and the image style reconstruction network is configured to process the original data object and reconstruct the original data object into a target data object, wherein the target data object maintains the content of the original data object and the content of the target data object has a target style.

If the original data object is original image data, the original image data may be input to an image style reconstruction network, which is configured to process the original image data and reconstruct the original image data into target image data (target data object).

If the original data object is original video data, each frame of original image data in the original video data can be extracted, each frame of original image data is respectively input into an image style reconstruction network, the image style reconstruction network structurally processes each frame of original image data, each frame of original image data is reconstructed into target image data, and each frame of target image data replaces the original image data in the original video data to obtain target video data (target data object).

In a specific implementation, as shown in fig. 2, a task queue 220 may be configured in the memory for each instance 211 of the generative countermeasure network 210, such that each instance 211 of the generative countermeasure network 210 is configured with a task queue 220, and the task queue 220 is used for storing a stylized task that is stylized by applying each instance 211 of the generative countermeasure network 210.

Under the condition that resources are sufficient, stylized tasks in the task queue can unify image styles.

In the case of resource shortage, the stylized task in the task queue may contain at least two image styles.

Typically, the task queue is a FIFO (First in First out) queue, so that the instances that generate the countermeasure network execute the stylized tasks in chronological order.

In some cases, the sequence of the stylized tasks can be scheduled according to factors such as the priority of the stylized tasks, the estimated time consumption of the stylized tasks and the like, so that the efficiency of generating the examples of the countermeasure network to execute the stylized tasks is improved, particularly, the stylized tasks with the same image stylized in a certain range are aggregated under the condition of resource shortage, the operation of switching the configuration files is reduced, and the frequency of switching the configuration files is reduced.

Then, as shown in fig. 2, a stylized task is created for the current raw data object 230, i.e., the stylized task has an identification (e.g., ID, address, etc.) therein that points to the raw data object, in order to read the raw data object itself.

And inquiring a task queue configured for an instance corresponding to the image style reconstruction network in all task queues to serve as a target queue, so that the stylized task is written into the target queue to wait for execution.

For the image style reconstruction network, the stylization tasks are read from the target queue in a predetermined order and executed to input the raw data object 230 into the instance 211 corresponding to the generation countermeasure network 210 (i.e., the image style reconstruction network) for reconstruction as the target data object 240.

For some controllable generation countermeasure networks, the stylized task may write stylized parameters, such as strength of stylization, etc., that control the generation of the countermeasure network in addition to the identification (such as ID, address, etc.) that points to the original data object.

Then, when the stylized task is executed, the original data object and the stylized parameters can be read from the stylized task, the original data object is input into an instance corresponding to the image style reconstruction network, and the original data object is reconstructed into the target data object according to the stylized parameters, so that the flexibility of stylization is improved.

In this embodiment, a generative countermeasure network of a fixed structure is loaded; inquiring and generating a plurality of configuration files which are all adaptive to the countermeasure network, wherein each configuration file records parameters which are obtained by training the countermeasure network and take a certain image style as a target for reconstruction; loading each configuration file respectively for generating the countermeasure network; receiving a stylization request sent by a first client for calling a stylization service for an original data object; responding to the stylization request, and determining the image style of the original data object to be reconstructed as a target style; searching a generated countermeasure network of a loaded target file as an image style reconstruction network, wherein the target file is a configuration file recorded with parameters obtained by training the generated countermeasure network by taking a reconstructed target style as a target; the original data object is input into an image style reconstruction network and reconstructed into a target data object, and the target data object maintains the content of the original data object and has a target style. The embodiment provides a generation countermeasure network with a unified structure to realize various image styles, a countermeasure mechanism in the generation countermeasure network has universality, a discriminator is used for learning and judging the image style of image data, the operations of designing an objective function for measuring the stylization quality and the like can be reduced, when an image style is newly added, the project of the prior image style can be inherited, the repeated development work is reduced, the technical threshold of development is greatly reduced, the development workload is reduced, the development time consumption is reduced, the development timeliness is improved, in the process of continuously accumulating the image style, the structure of the generation countermeasure network is fixed into one, configuration files are mainly accumulated, the management is convenient, the expansibility is strong, the stylization can be realized by loading the configuration files into the generation countermeasure network as required, the occupation of storage resources can be greatly reduced, and the burden on resources such as a processor, a memory and the like can be reduced during the operation.

Example two

Fig. 3 is a flowchart of a document stylization method according to a second embodiment of the present invention, in which post-processing operations are added to the second embodiment. As shown in fig. 3, the method includes:

and 301, loading the generation countermeasure network of the fixed structure.

Step 302, inquiring and generating multiple configuration files adaptive to the countermeasure network.

And parameters obtained by generating the antagonistic network training by taking a certain image style as a target are recorded in each configuration file.

And step 303, respectively loading each configuration file to generate the countermeasure network.

And step 304, receiving a stylization request sent by the first client for calling the stylization service for the original data object.

Step 305, in response to the stylization request, determining the image style of the original data object to be reconstructed as the target style.

And step 306, searching the generation countermeasure network of the loaded target file as an image style reconstruction network.

The target file is a configuration file recorded with parameters obtained by generating the antagonistic network training by taking the reconstructed target style as a target.

And 307, inputting the original data object into an image style reconstruction network to be reconstructed into a target data object.

Wherein the target data object maintains the content of the original data object and has a target style.

Step 308, load image processing programs configured for one or more image styles.

In practical applications, the generation countermeasure network may reconstruct the image style as a whole, but has a lack of details of some image styles, for which case, the embodiment may configure some image styles with image processing programs, and these image processing programs may implement one or more functions, and these functions usually perform image processing on the basis of pixel points of image data, such as face detection, edge detection, OCR (optical character recognition), and so on.

These image processing programs may be packaged in files that are independent of the generation of the countermeasure network, such as DLL (Dynamic Link Library) files.

Different image styles may use the same image processing program, so the functions implemented by the image processing program may be divided according to their multiplexing degree, which is not limited in this embodiment.

Generally, the smaller the granularity of the function realized by dividing the image processing program is, the more beneficial the function is to be multiplexed by different image styles, especially by the image style which is possibly newly added in the future, and the development cost can be effectively reduced.

Step 309, inquiring an image processing program matched with the target style as a target processing program.

In this embodiment, the technician has configured one or more image processing programs for each image style in advance according to the service adaptation degree, and each image processing program has a calling sequence therebetween.

As shown in fig. 2, in determining the target style, one or more image handlers configured for the target style may be queried as target handler 250.

Step 310, calling a target processing program to execute post-processing on the target data object.

In the present embodiment, as shown in fig. 2, the target processing program may be called in order of the calling order of the target processing program 250 to execute image processing on the target data object, the image processing belonging to post-processing with respect to the reconstructed image style.

The input (object data) of the first object handler is the output (object data) of the stylized network, the input (object data) of the non-first object handler is the output (object data) of the previous object handler, and the last object handler outputs the final object data.

If the target data object is target image data, the target processing program can be called in sequence according to the calling sequence of the target processing program to execute post-processing on the target image data.

If the target data object is target video data, each frame of target image data in the target video data can be extracted, the target processing programs are sequentially called according to the calling sequence of the target processing programs to execute post-processing on the target image data, and the target image data after each frame of post-processing is replaced with the target image data before post-processing in the target video data.

For different service requirements, post-processing combined by different target processing programs is different, and this embodiment does not limit this.

In one example, a target element may be detected in the original data object, wherein the target element is not suitable for being reconstructed into a target style, and the effect of the target element expressed on the screen is not suitable for the business, so that any one of the following adjustments is performed on the target element:

1. the target element is mapped back into the target data object.

It is considered that the styles of the reconstructed image generally do not cause obvious position migration to the content in the original data object, and some image styles cause obvious blurring to some content, especially to some small-volume content, such as face data, subtitle data, and the like, which can be used as target elements.

In this adjustment, the target element may be used as a mask, and the candidate element located under the mask may be detected in the target data object, that is, the candidate element and the target element may be located at the same position.

2. The content located within the target element is deleted in the target data object.

Some image styles may construct redundant data on some content, which affects the content, for example, for sketch, drawing and other image styles, some redundant lines may be generated on the face data, that is, lines located inside the face data and outside five sense organs, and the content may be used as a target element.

In this adjustment, the target element is used as a mask, and the candidate element located under the mask is detected in the target data object, that is, the candidate element is located at the same position as the target element.

3. The target element is fused into the target data object.

Considering that the styles of the reconstructed image generally do not cause obvious position migration to the content in the original data object, and some image styles may cause a certain blur to some content, such as face data, etc., the content may be used as a target element.

In this adjustment, the target element may be used as a mask, and the candidate element located under the mask is detected in the target data object, that is, the candidate element and the target element are located at the same position, at this time, the target element is fused to the candidate element in the target data object, that is, the color of the pixel point of the target element and the color of the pixel point of the candidate element are fused in a linear (i.e., weighted sum) manner, and the definition of the target element and the candidate element is improved while a certain image style is maintained.

In another example, because some samples have more complex image styles, some styles occupy dominant positions (i.e., main styles), the attention of the user can be focused on the main styles in a short time, some image styles occupy details and occupy non-dominant positions (i.e., non-main styles), but the user can also focus on the non-main styles in some times, for example, the main style of a certain cure series of mobile phone games is a warm-colored (e.g., light yellow green) oil painting style, each element in the mobile phone games is designed more delicately, a feeling of warm and mild cure can be embodied, and the non-main styles can be embodied in the aspects of obvious edges, brown color, more rounded light and shadow changes, and the like.

The learning difficulty of the main body style is low, the learning difficulty of the non-main body style is high, and the generation countermeasure network for the image style training can realize the reconstruction of the main body style in the image style.

In this example, the target style may be characterized by a first style feature, a second style feature, the first style feature being a subject style of the target style, and the second style feature being a non-subject style of the target style.

Then, if the target data object has the first style feature, the target processing program may be sequentially called according to the calling order of the target processing program to perform image processing on the target data object, so as to add the second style feature to the target data object, so that the target data object has a complete target style.

Of course, the above post-processing is only an example, and when the embodiment is implemented, other post-processing may be set according to actual situations, which is not limited in the embodiment. In addition to the above post-processing, those skilled in the art may also adopt other post-processing according to actual needs, and this embodiment is not limited to this.

EXAMPLE III

Fig. 4 is a flowchart of a file stylization method according to a second embodiment of the present invention, where the second embodiment is added with business operations based on the foregoing embodiment. As shown in fig. 4, the method includes:

step 401, loading the generation countermeasure network of the fixed structure.

Step 402, inquiring and generating multiple configuration files adaptive to the countermeasure network.

And step 403, loading each configuration file for generating the countermeasure network respectively.

Step 404, receiving a stylization request sent by a first client for calling a stylization service for an original data object.

Step 405, querying user information of the user logged in the first client.

And step 406, checking the legality of the stylized request according to the user information.

The method comprises the steps that a user who calls a stylized service of a public network has certain authority limit, so that when a stylized request sent by a first client calling the stylized service is received, user identification (such as ID) of the user who logs in at the first client at present can be read from the stylized request, and the user information of the user is carried in a database according to the user identification, so that the user information is used for checking the legality of the stylized request, the safety of the stylized service is guaranteed, and the normal development of the stylized service is guaranteed.

In one example, in the stage of popularizing the stylized service, in order to reduce the number of times that the user calls the stylized service and reduce the consumption of resources, the number of times of trying the stylized service may be configured for the user according to information such as the level of the user, and the stylized object during trying is limited to be image data.

Then, in this example, the number of times the user has left to try the stylized service may be read from the user information and the format of the original data object detected.

And if the number of the remaining trial stylization services is greater than 0 and the format of the original data object is image data, determining that the legality of the stylization request is legal.

And if the number of the remaining trial stylization services is equal to 0 or the format of the original data object is video data, determining that the legality of the stylization request is illegal.

In another example, the user may subscribe to the stylized service by directly purchasing the stylized service, purchasing a member, etc., and the stylized service is used by subscribing to the stylized service according to the times and/or the time, then, in this example, the total times and/or the time range of the users subscribing to the stylized service, the times of the users cumulatively calling the stylized service may be read from the user information;

and if the number of times of calling the stylized service is less than the total number of times and/or the current time is within the time range, determining that the legality of the stylized request is legal.

And if the accumulated times of calling the stylized service are equal to the total times and/or the current time is positioned outside the time range, determining that the legality of the stylized request is illegal.

In yet another example, the number of frames of the original data object may be counted, and the number of frames may be multiplied by the preset unit rights data, thereby mapping the number of frames to the first rights data consumed to invoke the stylized service.

And reading second authority data which the user has from the user information, and comparing the first authority data with the second authority data.

And if the second permission data is larger than or equal to the first permission data, determining that the legality of the stylized request is legal.

And if the second authority data is smaller than the first authority data, determining that the legality of the stylized request is illegal.

The unit authority data, the first authority data and the second authority data are all virtual authority data, such as virtual value-added data provided by a website.

Of course, the above-mentioned manner of verifying the validity is only an example, and when the embodiment is implemented, other manners of verifying the validity may be set according to actual situations, which is not limited in this embodiment. In addition, besides the above-mentioned way of verifying the validity, a person skilled in the art may also adopt other ways of verifying the validity according to actual needs, which is not limited in this embodiment.

And 407, if the stylization request is legal, responding to the stylization request, and determining the image style of the original data object to be reconstructed as a target style.

If the stylized request of the first client is validated, the raw object data may be stylized in response to the stylized request.

And if the stylization request of the first client is confirmed to be illegal, generating prompt information, sending the prompt information to the first client for displaying, and prompting the user that the calling of the stylization service is illegal and proposing the reason of the illegal, wherein at the moment, the stylization of the original object data is forbidden without responding to the stylization request.

And step 408, searching the generation countermeasure network of the loaded target file as an image style reconstruction network.

And step 409, inputting the original data object into an image style reconstruction network to be reconstructed into a target data object.

The target data object maintains the content of the original data object and has a target style.

And step 410, if the original data object is video data of a content introduction game, adding promotion elements related to the game in the target data object.

In this embodiment, if the format of the original data object is video data and the purpose of the service is to introduce a game into the content, a promotion element related to the game may be added to the target data object, where the promotion element is data for promoting the game, for example, LOGO (icon), banner (Banner information), EC (ending section), and the like of a platform for distributing the game.

And 411, publishing the target data object added with the promotion element in a specified channel so as to push the target data object to a second client for playing when the second client accesses the channel.

Under the condition that the promotion elements are added to the target data objects, the target data objects can be published in a specified channel (such as news information, short videos, novel reading, sports health and the like), when a second client accesses the channel, the target data objects with the added promotion elements can be pushed to the second client, the second client plays the target data objects with the added promotion elements, and if a user generates interest in the game in the process of browsing the target data objects with the added promotion elements, the game can be searched and downloaded from a game distribution platform according to the promotion elements.

Example four

Fig. 5 is a schematic structural diagram of a document formatting device according to a fourth embodiment of the present invention. As shown in fig. 5, the apparatus includes:

a generation countermeasure network loading module 501 for loading a generation countermeasure network of a fixed structure;

a configuration file query module 502, configured to query a plurality of configuration files adapted to the generated countermeasure network, where each of the configuration files records parameters obtained by training the generated countermeasure network with a goal of reconstructing a certain image style;

a configuration file loading module 503, configured to load each configuration file for the generated countermeasure network;

a stylized request receiving module 504, configured to receive a stylized request sent by a first client invoking a stylized service for an original data object;

a target style determining module 505, configured to determine, as a target style, an image style to be reconstructed of the original data object in response to the stylization request;

an image style reconstruction network searching module 506, configured to search the generated countermeasure network loaded with a target file as an image style reconstruction network, where the target file is the configuration file recorded with parameters obtained by training the generated countermeasure network with the reconstructed target style as a target;

a target data object reconstruction module 507, configured to input the original data object into the image style reconstruction network to reconstruct the original data object into a target data object, where the target data object maintains the content of the original data object and has the target style.

In an embodiment of the present invention, the configuration file loading module 503 is further configured to:

generating a plurality of instances for the generating a countermeasure network, the number of instances being the same as the number of profiles;

and loading the parameters in each configuration file into each instance respectively.

creating a plurality of processes, wherein the number of the processes is the same as that of the configuration files;

an instance is created for the generative confrontation network in each of the processes.

In one embodiment of the invention, each of the instances is configured with a task queue;

the target data object reconstruction module 507 is further configured to:

creating a stylized task for the raw data object;

querying the task queue configured for the instance corresponding to the image style reconstruction network as a target queue;

writing the stylized task into the target queue;

reading the stylized tasks from the target queue in sequence;

and executing the stylization task to input the original data object into the instance corresponding to the image style reconstruction network to reconstruct the original data object into a target data object.

In an embodiment of the present invention, the target data object reconstruction module 507 is further configured to:

reading the original data object and the stylized parameters from the stylized task;

and inputting the original data object into the example corresponding to the image style reconstruction network, and reconstructing the original data object into a target data object according to the stylized parameters.

In another embodiment of the present invention, the configuration file loading module 503 is further configured to:

generating instances for the generating a countermeasure network, the number of instances being less than the number of profiles;

inquiring the style of the next image to be reconstructed as a sample style aiming at the current example;

and if a certain configuration file records parameters obtained by training the generated countermeasure network by taking the sample style as a reconstructed target, loading the configuration file into the example.

In one embodiment of the present invention, further comprising:

the image processing program loading module is used for loading an image processing program configured for one or more image styles;

the target processing program query module is used for querying the image processing program matched with the target style to serve as a target processing program;

and the post-processing execution module is used for calling the target processing program to execute post-processing on the target data object.

In an embodiment of the present invention, the post-processing execution module is further configured to:

detecting a target element in the raw data object, the target element not being suitable for reconstruction into the target style;

performing any of the following adjustments to the target element:

detecting a candidate element located under the mask in the target data object with the target element as a mask, the target element replacing the candidate element in the target data object;

detecting candidate elements positioned under the mask in the target data object by taking the target element as the mask, deleting data positioned in the target element in the target data object, and if the deletion is completed, overlaying the target data object on the original data object to be used as a new target data object;

detecting, in the target data object, a candidate element located under the mask with the target element as a mask, and fusing, in the target data object, the target element to the candidate element.

In another embodiment of the present invention, the target style is characterized by a first style characteristic and a second style characteristic, and the post-processing execution module is further configured to:

and if the target data object has the first style characteristic, calling the target processing program to execute image processing on the target data object so as to add the second style characteristic to the target data object.

In one embodiment of the present invention, further comprising:

the user information inquiry module is used for inquiring the user information of the user logged in the first client;

the legality checking module is used for checking the legality of the stylized request according to the user information; if the stylization request is legal, the target style determination module 505 is invoked.

In an embodiment of the present invention, the validity checking module is further configured to:

reading the number of times that the user remains to try the stylized service from the user information;

if the number of times of trying the stylized service is more than 0 and the original data object is image data, determining that the stylized request is legal;

or,

reading the total times and/or time range of the user subscribing the stylized service and the times of the user cumulatively calling the stylized service from the user information;

if the number of times of calling the stylized service is less than the total number of times and/or the current time is within the time range, determining that the stylized request is legal;

or,

counting the frame number of the original data object;

mapping the frame number to first permission data consumed by calling the stylized service

Reading second authority data which the user has from the user information;

and if the second permission data is larger than or equal to the first permission data, determining that the stylized request is legal.

In one embodiment of the present invention, further comprising:

the promotion element adding module is used for adding promotion elements related to the game into the target data object if the original data object is video data of a content introduction game;

and the target data object publishing module is used for publishing the target data object added with the promotion element in a specified channel so as to push the target data object to a second client side for playing when the second client side accesses the channel.

The file stylization device provided by the embodiment of the invention can execute the file stylization method provided by any embodiment of the invention, and has the corresponding functional modules and beneficial effects of executing the file stylization method.

EXAMPLE five

FIG. 6 illustrates a block diagram of an electronic device 10 that may be used to implement an embodiment of the invention. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed herein.

As shown in fig. 6, the electronic device 10 includes at least one processor 11, and a memory communicatively connected to the at least one processor 11, such as a Read Only Memory (ROM) 12, a Random Access Memory (RAM) 13, and the like, wherein the memory stores a computer program executable by the at least one processor, and the processor 11 can perform various suitable actions and processes according to the computer program stored in the Read Only Memory (ROM) 12 or the computer program loaded from a storage unit 18 into the Random Access Memory (RAM) 13. In the RAM 13, various programs and data necessary for the operation of the electronic apparatus 10 may also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other via a bus 14. An input/output (I/O) interface 15 is also connected to bus 14.

A number of components in the electronic device 10 are connected to the I/O interface 15, including: an input unit 16 such as a keyboard, a mouse, or the like; an output unit 17 such as various types of displays, speakers, and the like; a storage unit 18 such as a magnetic disk, an optical disk, or the like; and a communication unit 19 such as a network card, modem, wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.

Processor 11 may be a variety of general and/or special purpose processing components having processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various specialized Artificial Intelligence (AI) computing chips, various processors running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, or the like. The processor 11 performs the various methods and processes described above, such as the file stylization method.

In some embodiments, the file stylization method may be implemented as a computer program tangibly embodied on a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and/or installed onto the electronic device 10 via the ROM 12 and/or the communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the file stylization method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the file stylization method by any other suitable means (e.g., by means of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

A computer program for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions/acts specified in the flowchart and/or block diagram block or blocks to be performed. A computer program can execute entirely on a machine, partly on a machine, as a stand-alone software package partly on a machine and partly on a remote machine or entirely on a remote machine or server.

In the context of the present invention, a computer-readable storage medium may be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer readable storage medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, the computer readable storage medium may be a machine readable signal medium. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the electronic device. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), blockchain networks, and the internet.

The computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, so that the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service are overcome.

EXAMPLE six

Embodiments of the present invention further provide a computer program product, which includes a computer program, and when executed by a processor, the computer program implements the file stylization method as provided in any one of the embodiments of the present invention.

Computer program product in implementing the computer program code for carrying out operations of the present invention may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present invention may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solution of the present invention can be achieved.

The above-described embodiments should not be construed as limiting the scope of the invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method of stylizing a document, comprising:

loading a generation countermeasure network of a fixed structure;

searching the generated confrontation network loaded with a target file as an image style reconstruction network, wherein the target file is the configuration file recorded with parameters obtained by training the generated confrontation network by taking the reconstructed target style as a target;

inputting the original data object into the image style reconstruction network to reconstruct the original data object into a target data object, wherein the target data object maintains the content of the original data object and has the target style;

wherein the loading each of the configuration files for the generated countermeasure network comprises:

loading parameters in each configuration file into each instance respectively;

each instance is configured with a task queue; inputting the original data object into the image style reconstruction network to be reconstructed into a target data object, wherein the reconstructing comprises:

creating a stylized task for the raw data object;

writing the stylized task into the target queue;

reading the stylized tasks from the target queue according to the sequence;

2. The method of claim 1, wherein generating the plurality of instances for the generative countermeasure network comprises:

3. The method of claim 1, wherein said performing the stylization task to input the raw data object into the instance corresponding to the image-style reconstruction network for reconstruction as a target data object comprises:

4. The method of claim 1, wherein said loading each of said configuration files separately for said generating a countermeasure network comprises:

and if a certain configuration file records a parameter obtained by training the generated confrontation network by taking the sample style as a reconstructed target, loading the configuration file into the example.

5. The method of any of claims 1-2, 4, further comprising:

loading an image processing program configured for one or more of the image styles;

inquiring the image processing program matched with the target style as a target processing program;

and calling the target processing program to execute post-processing on the target data object.

6. The method of claim 5, wherein invoking the target handler to perform post-processing on the target data object comprises:

performing any of the following adjustments to the target element:

7. The method of claim 5, wherein the target style is characterized by a first style characteristic and a second style characteristic, and wherein invoking the target handler to perform post-processing on the target data object comprises:

8. The method according to any of claims 1-2 and 4, wherein after receiving the stylized request sent by the first client invoking the stylized service for the original data object, the method further comprises:

querying user information of a user logged in the first client;

verifying the legality of the stylized request according to the user information; and if the stylized request is legal, executing the stylized request, and determining the style to be reconstructed of the original data object as a target style.

9. The method of claim 8, wherein said verifying the legitimacy of the stylized request based on the user information comprises:

if the number of remaining trial of the stylized service is greater than 0 and the original data object is image data, determining that the stylized request is legal;

or,

counting the frame number of the original data object;

Reading second authority data which the user has from the user information;

10. The method of any of claims 1-2, 4, further comprising:

if the original data object is video data of a content introduction game, adding promotion elements related to the game in the target data object;

and issuing the target data object added with the promotion element in a specified channel so as to push the target data object to a second client side for playing when the second client side accesses the channel.

11. A document stylizing apparatus, comprising:

the configuration file query module is used for querying a plurality of configuration files which are matched with the generated confrontation network, and each configuration file records parameters which are obtained by training the generated confrontation network and take reconstruction of a certain image style as a target;

a target data object reconstruction module, configured to input the original data object into the image style reconstruction network and reconstruct the original data object into a target data object, where the target data object maintains the content of the original data object and has the target style;

wherein the configuration file loading module is further configured to:

loading parameters in each configuration file into each instance respectively;

each instance is configured with a task queue; the target data object reconstruction module is further to:

creating a stylized task for the raw data object;

writing the stylized task into the target queue;

reading the stylized tasks from the target queue in sequence;

12. An electronic device, characterized in that the electronic device comprises:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

the memory stores a computer program executable by the at least one processor to enable the at least one processor to perform the file stylization method of any one of claims 1-10.

13. A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program for causing a processor to carry out the file stylization method of any one of claims 1-10 when executed.