CN118691767A - Image processing method, system, electronic device and computer program product - Google Patents

Abstract

The present application discloses an image processing method, system, electronic device and computer program product, which relate to large model technology and image processing fields. The method may include: identifying a product image to be processed, wherein the image content of the product image includes at least one product to be displayed; obtaining text description information corresponding to the product to be displayed in the product image, wherein the text description information is used to at least describe the wearing performance result displayed by the product to be displayed when it is worn by a carrier; using the text description information, guiding the image processing model to analyze the product image to obtain a target image, wherein the image processing model is obtained by training a Wensheng graph model, and the image content of the target image is used to simulate the wearing performance result. The present application solves the technical problem of poor effect of simulated wearing performance results in images.

Description

Image processing method, system, electronic device and computer program product

Technical Field

The present application relates to large model technology and image processing fields, and more specifically, to an image processing method, system, electronic device and computer program product.

Background Art

At present, with the rapid development of e-commerce, users' demand for online shopping experience is growing. For example, they hope to have a more intuitive understanding of the wearing effect of a product before purchasing it. Therefore, the technology of using computer vision and image processing technology to merge product images with model graphics to simulate the actual wearing effect has emerged, becoming an important means to improve shopping experience and increase customer satisfaction, as well as an important trend in the e-commerce industry. Virtual fitting technology can.

In the related art, if you need to simulate the wearing effect of a certain product on a certain model, you usually need a specific model picture, which poses certain challenges to some e-commerce merchants. For example, using existing model pictures may involve copyright issues, the wearing effect is limited by the model's original body shape, and the items or clothes worn by the model may interfere with the fitting effect. In general, the services for displaying the wearing effect of products in images in the related art are greatly limited, and customized selection of wearing effects cannot be achieved. Therefore, there is still a technical problem that the simulated wearing performance results in the image are poor.

To address the above-mentioned problems, no effective solution has been proposed yet.

Summary of the invention

The embodiments of the present application provide an image processing method, system, electronic device, and computer program product to at least solve the technical problem of poor results of simulated wearing performance in images.

According to one aspect of an embodiment of the present application, a method for processing an image is provided. The method may include: identifying a product image to be processed, wherein the image content of the product image includes at least one product to be displayed; obtaining text description information corresponding to the product to be displayed in the product image, wherein the text description information is used to at least describe the wearing performance result displayed by the product to be displayed when it is worn by a carrier; using the text description information, guiding an image processing model to analyze the product image to obtain a target image, wherein the image processing model is obtained by training a text image model, and the image content of the target image is used to simulate the wearing performance result.

According to another aspect of the embodiment of the present application, a method for processing clothing images is provided. The method may include: identifying a virtual clothing store deployed on an e-commerce platform, and identifying a clothing product image to be processed from the virtual clothing store, wherein the image content of the clothing product image includes at least one clothing product to be displayed; obtaining text description information corresponding to the clothing product to be displayed in the clothing product image, wherein the text description information is used to at least describe the wearing performance result displayed when the clothing product to be displayed is worn by a carrier; using the text description information, guiding the image processing model to analyze the clothing product image to obtain a target image, wherein the image processing model is obtained by training the text image model, and the image content of the target image is used to simulate the wearing performance result; sending the target image to the e-commerce platform.

According to another aspect of the embodiment of the present application, another image processing method is provided. The method may include: in response to an image input operation on an operation interface, displaying a product image to be processed on the operation interface, wherein the image content of the product image includes at least one product to be displayed; in response to a text input operation on the operation interface, displaying text description information corresponding to the product to be displayed in the product image on the operation interface, wherein the text description information is used to at least describe the wearing performance result displayed by the product to be displayed when it is worn by a carrier; in response to an image generation operation on the operation interface, displaying a target image matching the product image and the text description information on the operation interface, wherein the target image is obtained by using the text description information to guide the image processing model to analyze the product image, the image processing model is obtained by training the text image model, and the image content of the target image is used to simulate the wearing performance result.

According to another aspect of the embodiment of the present application, another image processing method is provided. The method may include: displaying a product image to be processed on a presentation screen of a virtual reality VR device or an augmented reality AR device, wherein the image content of the product image includes at least one product to be displayed; obtaining text description information corresponding to the product to be displayed in the product image, wherein the text description information is used to at least describe the wearing performance result displayed by the product to be displayed when it is worn by a carrier; driving the VR device or AR device to use the text description information to guide the image processing model to analyze the product image to obtain a target image, wherein the image processing model is obtained by training the text image model, and the image content of the target image is used to simulate the wearing performance result; displaying the target image on the presentation screen of the VR device or AR device.

According to another aspect of this embodiment of the present application, an image processing system is provided. The system may include: a client, for uploading a product image to be processed, wherein the image content of the product image includes at least one product to be displayed; uploading text description information corresponding to the product to be displayed in the product image, wherein the text description information is used to at least describe the wearing performance result displayed by the product to be displayed when it is worn by a carrier; a server, for using the text description information to guide the image processing model to analyze the product image to obtain a target image, wherein the image processing model is obtained by training a text image model, and the image content of the target image is used to simulate the wearing performance result; and sending the target image to the client.

According to another aspect of an embodiment of the present application, an electronic device is also provided. The electronic device may include a memory and a processor: the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions. When the above-mentioned computer-executable instructions are executed by the processor, a method for processing an image of any one of the above-mentioned items is implemented.

According to another aspect of an embodiment of the present application, a processor is further provided, and the processor is used to run a program, wherein any one of the above-mentioned image processing methods is executed when the program is running.

According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute any one of the above-mentioned image processing methods.

According to another aspect of the embodiment of the present application, a computer program product is also provided. The computer program product includes a computer program, and when the computer program is executed by a processor, the image processing method of the embodiment of the present application is implemented.

In an embodiment of the present application, if it is necessary to determine and display the wearing performance of a certain product on a carrier, the product image to be processed of the product can be identified. The text description information of the product displayed on the carrier is obtained. The text description information can be used to guide the image processing model pre-trained by the Wensheng graph model to analyze the product image and obtain a target image that can simulate the wearing performance of the product on the carrier. Since the embodiment of the present application takes into account the problems in the relevant technology, a virtual fitting technology using the Wensheng graph framework is proposed. The virtual fitting effect of the product is expressed in words through a simple text description, and the image processing model trained by the Wensheng graph model is used to analyze it, so that a highly realistic model fitting effect image can be generated, thereby eliminating the dependence on specific model pictures, providing users with a product display effect with greater freedom, flexibility and customized selection, and achieving the purpose of reducing the limitations of the wearing performance results of the product simulated by the image, thereby achieving the technical effect of improving the effect of the simulated wearing performance results in the image, and solving the technical problem of poor effect of the simulated wearing performance results in the image.

It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

FIG1 is a schematic diagram of an application scenario of an image processing method according to an embodiment of the present application;

FIG2 is a flow chart of an image processing method according to an embodiment of the present application;

FIG3 is a flow chart of another image processing method according to an embodiment of the present application;

FIG4 is a flow chart of another image processing method according to an embodiment of the present application;

FIG5 is a flow chart of another image processing method according to an embodiment of the present application;

FIG6 is a schematic diagram of an image processing system according to an embodiment of the present application;

FIG7 is a schematic diagram of a virtual try-on process system based on a Wensheng graph framework according to an embodiment of the present application;

FIG8 is a schematic diagram of a knowledge injection position according to an embodiment of the present application;

FIG9 is a schematic diagram of a knowledge injection method according to an embodiment of the present application;

FIG10 is a schematic diagram of image processing by a computer device according to an embodiment of the present invention;

FIG11 is a schematic diagram of an image processing device according to an embodiment of the present application;

FIG12 is a schematic diagram of a device for processing clothing images according to an embodiment of the present application;

FIG13 is a schematic diagram of another image processing device according to an embodiment of the present application;

FIG14 is a schematic diagram of another image processing device according to an embodiment of the present application;

FIG15 is a structural block diagram of a computer terminal according to an embodiment of the present application;

FIG16 is a block diagram of an electronic device for an image processing method according to an embodiment of the present application;

FIG17 is a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image processing method according to an embodiment of the present application;

FIG. 18 is a structural block diagram of a computing environment of an image processing method according to an embodiment of the present application.

DETAILED DESCRIPTION

In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

The technical solution provided in this application is mainly implemented by large-scale model technology. The large model here refers to a deep learning model with large-scale model parameters, which can usually contain hundreds of millions, tens of billions, hundreds of billions, trillions or even more than ten trillion model parameters. The large model can also be called a foundation model/foundation model. The large model is pre-trained by large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as a large-scale language model (Language Model for Information Retrieval, referred to as LLM), a multi-modal pre-training model, etc.

It should be noted that when the large model is actually applied, the pre-trained model can be fine-tuned through a small number of samples so that the large model can be applied to different tasks. For example, the large model can be widely used in natural language processing (NLP), computer vision, speech processing and other fields, and can be specifically applied to computer vision tasks such as visual question answering (VQA), image description (IC), image generation, etc., and can also be widely used in text-based sentiment classification, text summary generation, machine translation and other natural language processing tasks. Therefore, the main application scenarios of the large model include but are not limited to digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc. In the embodiment of the present application, the target language processing model obtained by training the generative large model in the dialogue scenario is used as an example to explain the application program generation.

It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

The Stable Diffusion (SD) model is a diffusion model and a generative model. Diffusion models are usually used to describe the propagation process of information, disease propagation, etc. in the network. Generative models refer to a type of machine learning model used to model data. The Stable Diffusion model can be used to model the propagation and diffusion process of certain information or phenomena in the network;

Variational Autoencoder (VAE) is an image encoding network used to extract image features. VAE is a generative model that is often used to learn the potential representation of data. In this description, it is one of the basic building blocks of the Stable Diffus ion model, which is used to process the extraction of image features. This illustrates the importance and application value of VAE in generative models;

U-Net (Unet for short) is a convolutional neural network structure, which is usually used for image segmentation tasks. It has a downsampling module and an upsampling module, which can effectively capture local and global information in the image. The Unet network is one of the basic components of the Stable Diffus ion model, which shows that the Unet network can be applied to the construction of the Stable Diffus ion model to process feature extraction and segmentation of image data. This shows the importance of the Unet network in image processing and generative models;

Virtual TryOn technology, the input information is an image containing a portrait and a flat target clothing image. The goal of this technology is to remove the original clothing of the portrait and fuse the target clothing image onto the portrait in a natural way. This technology may involve computer vision and graphics processing technologies such as image segmentation, clothing synthesis and image fusion to achieve a virtual display of the portrait trying on clothing;

Contrastive Language-Image Pre-Training (CLIP) is a model for image-text relevance matching. This model aims to match images and texts for relevance, so that the model can understand the semantic relationship between images and texts. The CLIP model is designed to process image and text data simultaneously, thereby achieving more comprehensive semantic understanding and relevance matching.

Example 1

According to an embodiment of the present application, a method for processing an image is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

Considering the huge amount of model parameters of the large model and the limited computing resources of the mobile terminal, the processing method of the above-mentioned image provided in the embodiment of the present application can be applied to the application scenario shown in Figure 1, but is not limited thereto. In the application scenario shown in Figure 1, the large model is deployed in the server 30, which can be a cloud. The server 30 can be connected to one or more terminal devices 10 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks, wherein the terminal device 10 can be a client device, and the client device here can include but is not limited to: a smart phone, a tablet computer, a laptop computer, a PDA, a personal computer, a smart home device, a vehicle-mounted device, etc., and the client devices together constitute the client relative to the server. An operation interface for obtaining product images can be deployed on the graphical user interface on the client device, and the operation interface can be an operation interface of an e-commerce platform. The terminal device 10 can interact with the user through the graphical user interface to realize the call of the large model, and then realize the image processing method provided in the embodiment of the present application. Information exchange can be carried out between the server 30 and the terminal device 10 through the network 20. A large model may be deployed in the database 40 in the server. In the embodiment of the present application, the database 40 stores an image processing model trained by the Wensheng graph model, which can be used to call through the server 30 to simulate the wearing performance results.

In the embodiment of the present application, the system composed of the terminal device, the network and the server can perform the following steps: if the user needs to be shown the wearing performance result of a product, he can perform input or selection operations on the operation interface of the corresponding terminal device 10, so as to obtain the product image corresponding to the product for which the wearing performance result is to be shown. The terminal device 10 can send the product image to the server 30 through the network 20. The user can also input the displayed wearing performance result and the text description information corresponding to the carrier on the operation interface. The terminal device 10 can send the text description information to the server 30 through the network 20. After receiving the product image and text description information, the server 30 may execute the following steps: Step S102, identifying the product image to be processed, wherein the image content of the product image includes at least one product to be displayed; Step S104, obtaining text description information corresponding to the product to be displayed in the product image, wherein the text description information is used to at least describe the wearing performance result displayed by the product to be displayed when it is worn by the carrier; Step S106, using the text description information to guide the image processing model to analyze the product image to obtain a target image, wherein the image processing model is obtained by training the text image model, and the image content of the target image is used to simulate the wearing performance result.

The target image of the wearing performance result to be displayed can be output to the terminal device 10 through the network 20 to display the wearing performance result of the product on the carrier for the user to view.

Based on the technical problem that the wearing performance results simulated in images are poor, the present application proposes a virtual fitting technology using a Vincent graph framework, which uses simple text descriptions to express the virtual fitting effects of the product, and uses an image processing model trained by a Vincent graph model to analyze it, thereby generating highly realistic model fitting effect images, thereby eliminating the dependence on specific model images, and providing users with product display effects with greater freedom, flexibility and customized choices, thereby reducing the limitations of simulating the wearing performance results of products through images, thereby achieving the technical effect of improving the wearing performance results simulated in images, and solving the technical problem of poor wearing performance results simulated in images.

It should be noted that, when the operating resources of the client device can meet the deployment and operating conditions of the large model, the embodiments of the present application can be carried out in the client device.

In the above operating environment, the present application provides a method for processing an image as shown in FIG2. It should be noted that the method for processing an image of this embodiment can be executed by the mobile terminal of the embodiment shown in FIG1. FIG2 is a flow chart of a method for processing an image according to an embodiment of the present application. As shown in FIG2, the method may include the following steps:

Step S202: identifying the product image to be processed.

In the technical solution provided in the above step S202 of the present application, the products to be displayed can be products that need to display wearing effects in the e-commerce platform, for example, they can be clothing, accessories, shoes and bags, etc. Clothing can be tops, bottoms and skirts, etc., accessories can be hair accessories and jewelry, etc., shoes and bags can be shoes and backpacks, etc. It should be noted that the above products to be displayed are only examples, and no specific restrictions are made here. As long as the products need to display wearing effects, they are within the protection scope of the embodiments of the present application.

Optionally, the image content displayed in the product image may include at least one product to be displayed. If the product to be displayed is clothing, the product image is a clothing image, which can also be called a clothing tile image. The product image here corresponds to the product to be displayed one by one, and no further details are given here.

In this embodiment, the product image to be processed may be identified.

Optionally, after the product image to be processed is acquired, the product image may be identified to obtain the product to be displayed therein.

For example, if a user or merchant of an e-commerce platform needs to obtain the wearing effect of a certain product, they can upload the product image corresponding to the product to be displayed on the operation interface of the corresponding terminal device, or filter out the image of the product that needs to display the wearing effect from the large number of product images displayed on the operation interface as the product image. After determining the product image on the operation interface of the terminal device, the product image can be transmitted to the server through the network. The received product image can be identified in the server to determine the product that needs to be displayed for the wearing effect, wherein the user can be a customer or merchant of the e-commerce platform.

It should be noted that the above process and method of obtaining product images are only for illustration and are not specifically limited here. As long as the method can capture product images of the products to be displayed, it is within the protection scope of the embodiments of the present application.

Step S204: obtaining text description information corresponding to the product to be displayed in the product image.

In the technical solution provided in the above step S204 of the present application, the text description information can be used to at least describe the wearing performance result displayed when the product to be displayed is worn by the carrier. The carrier can be a person who presents the wearing effect of the product, for example, it can be a model, or it can be a customer who is to purchase the product. This is only for example, and no specific limitation is made to the carrier presenting the wearing effect. The wearing performance result can be the wearing effect of the product on the carrier, which can also be called a virtual try-on effect or an upper body effect. For example, if the carrier is a model, it can be the upper body effect of the model, which can also be called a model try-on effect. This is only for example, and no specific limitation is made to the wearing performance result. It can correspond to the carrier of the actual worn product, and will not be described one by one here.

Optionally, the text description information can be a text description of the appearance of the carrier of the wearable product, the posture when showing the wearing effect, and the wearing method of the product. For example, the text description information can be used to describe the appearance characteristics of the carrier, such as gender, weight, accessories, height, body shape and hairstyle, and can also describe the posture characteristics of the carrier in front standing, side standing or sitting, and can also be used to describe the wearing method of the product to be displayed, for example, it can be the wearing method of clothing, whether the clothing is open-necked, rolled up sleeves, and layered, etc. If the text description information is input by the user according to his own display requirements for the wearing effect, the constructed carrier is a customized model, the corresponding appearance characteristics are customized model attributes, and the wearing attributes constructed based on the user's own display requirements for the wearing effect are customized clothing attributes.

It should be noted that the content and feature types described in the above text description information are only examples and are not specifically limited here. They can be determined according to the actual requirements of the wearing effect of the product to be displayed. As long as it can describe how the carrier displays the wearing effect in a textual manner, as well as the process and text description information of the carrier itself, they are all within the protection scope supported by the embodiments of this application.

In this embodiment, after the product image to be processed is identified, text description information corresponding to the product to be displayed in the product image may be obtained.

For example, if a customer or merchant on an e-commerce platform needs to obtain the wearing effect of a certain product, after uploading the product image on the operation interface, they can enter the appearance characteristics of the carrier that can describe the wearable display of the product in the corresponding text input box on the operation interface, or the posture characteristics of the carrier when the product is worn and displayed, and the way the carrier wears the product. The text of the above-mentioned appearance characteristics, wearing methods and posture characteristics can be used as the text description information of the product to be displayed.

For another example, the text description information can be transmitted to the server through the network, so that when simulating the wearing performance results, the server can understand the user's needs for the wearing effect display.

It should be noted that the above process and method of obtaining text description information are for illustration only and are not specifically limited here. As long as the operations and processes can construct a virtual try-on effect through text description, they are within the protection scope of the embodiments of the present application.

In related virtual fitting technologies, a specific model image is usually required, which may pose a challenge to some businesses. This may limit the flexibility of businesses and customers when trying on clothes, especially for customers of businesses that want to provide a personalized fitting experience. The casual use of existing model photos may touch on copyright issues, which may limit the freedom of businesses to use specific model images. Restricted by the model's original body shape and the clothes they wear, this may affect the fitting effect, making the application of virtual fitting technology subject to certain restrictions.

In general, the relevant virtual fitting technology still has room for further development in overcoming these challenges, such as developing more flexible and free virtual fitting technology to better meet the needs of merchants and consumers. Therefore, there is still a technical problem of high limitations in showing the wearing performance of products.

However, in the embodiment of the present application, the user is required to input text descriptions, which will be used to specify the background of the generated image, the gender of the model, the hairstyle and other attributes. The text description provided by the user may include specific requirements for the background and model attributes of the virtual try-on image. This method provides users with a convenient way to customize and specify the attributes of the virtual try-on image, thereby achieving a more personalized try-on effect for the displayed products. In general, through the above method, users can be allowed to specify various attributes of the virtual try-on image through simple text descriptions, which provides users with greater flexibility and customization options.

Step S206: Use the text description information to guide the image processing model to analyze the product image to obtain the target image.

In the technical solution provided in the above step S206 of the present application, the image processing model can be obtained by training the Wensheng graph model, which can be used to generate an image of the product being worn and displayed in accordance with the description of the text description information, and can be a virtual try-on model of this embodiment of the present application, which can also be called a TryOn model, and the model can be a dual Unet architecture. The Wensheng graph model can be a diffusion model. The image content of the target image can be used to simulate the wearing performance result, and the target image can be an image obtained by the Wensheng graph through the image processing model based on the text description information, which can be used to display the virtual fitting result. If the product to be displayed is clothing, the target image can be a picture of the clothing on the carrier. The Wensheng graph model can be a Wensheng graph framework.

In this embodiment, after obtaining the text description information corresponding to the product to be displayed in the product image, the text description information can be used to guide the image processing model to analyze the product image to obtain the target image.

Optionally, after obtaining the product image and the corresponding text description information, an image processing model can be used to analyze the text description information and the product image, so as to construct a target image that meets the user's requirements for the wearing effect of the displayed product in the text description information.

For example, after the server receives the product image and text description information, it can call the image processing model deployed in the server's database to analyze the product image and text description information, thereby simulating the wearing performance results of the product to be displayed when it is worn by the carrier in accordance with the text description information, and generating the corresponding target image.

In an embodiment of the present application, a virtual try-on method based on a text-based image framework is proposed, through which the industry pain points of virtual try-on technology, namely, the reliance on and limitation of specific model pictures, can be solved. This method can not only ensure the high realism of the simulated try-on, making the try-on effect more realistic, but also can realize the construction of the model try-on effect through a simple text description by the user, greatly improving the convenience of operation. This method supports language description to personalize the background in the picture, the gender, body shape and even skin color characteristics of the model, thereby achieving the technical effect of improving the flexibility of personalized customization.

Through the above steps S202 to S206 of the present application, if it is necessary to determine and display the wearing performance of a certain product on a carrier, the product image to be processed of the product can be identified. Obtain the text description information for displaying the product on the carrier. The text description information can be used to guide the image processing model pre-trained by the Wensheng graph model to analyze the product image and obtain a target image that can simulate the wearing performance of the product on the carrier. Since the embodiment of the present application takes into account the problems in the related art, a virtual fitting technology using the Wensheng graph framework is proposed, and the virtual fitting effect of the product is expressed in words through a simple text description, and the image processing model trained by the Wensheng graph model is used to analyze it, so that a highly realistic model fitting effect image can be generated, thereby eliminating the dependence on specific model pictures, providing users with a product display effect with greater freedom, flexibility and customized selection, and achieving the purpose of reducing the limitations of the wearing performance results of the product simulated by the image, thereby achieving the technical effect of improving the effect of the simulated wearing performance results in the image, and solving the technical problem of poor effect of the simulated wearing performance results in the image.

The above method of this embodiment is further introduced below.

As an optional implementation, step S206, using the text description information to guide the image processing model to analyze the product image to obtain the target image, includes: using the image processing model to extract text features from the text description information; using the text features to guide the image processing model to analyze the product image to obtain the target image.

In this embodiment, in the process of using text description information to guide the image processing model to analyze the product image, the image processing model can be used to extract text features from the text description information, and the text features are used to guide the image processing model to analyze the product image, so as to obtain a target image that meets the wearing effect corresponding to the text description information, wherein the text feature can be a text field in the text description information that can reflect the user's demand for the product wearing effect, and the text field can be a model feature description, a model posture description, or a background description of the target image, etc. The model feature description can be "short", "long hair", etc.

It should be noted that the above-mentioned text features and corresponding feature descriptions are only for illustrative purposes and are not specifically limited here. As long as the text features in the text description information can reflect the user's demand for dressing effects and the target image showing the dressing effects, they are within the protection scope of the embodiments of the present application.

Optionally, the process of extracting text features from the text description information and using the text features to guide the TryOn model to determine the target image involves complex tasks of natural language processing and image processing.

Optionally, natural language processing is used to process the text description information to extract semantic and contextual features of the text description information. For example, the text description information can be encoded into a high-dimensional vector representation, and these vectors are text features. It should be noted that the above process and method for extracting text features from text description information are only for illustration and are not specifically limited here.

Optionally, the text features may include descriptions related to clothing, such as the color, style, material, etc. of the clothing, and may also include descriptions about the model, such as gender, body shape, hairstyle, etc. These text features can help guide the TryOn model to generate a virtual try-on effect that conforms to the text description. For example, if the text description information mentions the text feature of "blue dress", the image processing model can use this text feature to ensure that the generated virtual try-on image contains the wearing effect of the blue dress. This is only an example, and there is no specific limitation on the text description information and the displayed wearing effect.

Optionally, the above-mentioned text features can be used to guide the TryOn model to generate a virtual try-on effect that conforms to the text features, that is, the TryOn model can generate a target image of the corresponding wearing effect according to these text features, ensuring that the generated target image conforms to the user's text description.

As an optional implementation, an image processing model is used to extract text features from text description information, including: using a text feature extraction model in the image processing model to extract text features from the text description information; using the text features to guide the image processing model to analyze the product image to obtain a target image, including: using the text features to guide the attention processing model in the image processing model to perform attention processing on the product image to obtain the target image.

In this embodiment, in the process of extracting text features from text description information using an image processing model, a text feature body model in the image processing model can be used to extract text features from the text description information. In the process of using text features to guide the image processing model to analyze product images, the text features can be used to guide the attention processing model in the image processing model to perform attention processing on the product image, thereby obtaining a target image, wherein the text feature extraction model can be a text encoder. The attention processing model can include a cross attention module.

Optionally, the dual U-Net TryOn model obtains the features of the text description through a text encoder, and injects these features into the Cross Attention module of the dual U-Net TryOn model. This setting can ensure that the text description entered by the user can directly affect the final virtual try-on effect, so that the user's personalized needs and customization requirements can be fully considered.

Optionally, a text encoder generally refers to a model or algorithm that can convert text information into vectors or other mathematical representations. These vectors or mathematical representations can be used in tasks such as text classification, semantic understanding, and language generation. Optionally, the text encoder of this embodiment includes deep learning-based models such as BERT and GPT, which can convert text into high-dimensional semantic vector representations so that computers can better understand and process text information.

Optionally, Cross Attention refers to the model calculating the correlation between sequences when processing multiple input sequences in order to better capture the correlation information between sequences. The cross attention mechanism is often used in sequence-to-sequence tasks such as machine translation, text summarization, etc. to help the model better understand the relationship between input sequences and generate more accurate outputs.

As an optional implementation, the method also includes: detecting the product image to obtain attribute information of the product to be displayed in the product image; using text features to guide the attention processing model in the image processing model to perform attention processing on the product image to obtain a target image, including: using text features to guide the attention processing model to perform attention processing on the attribute information of the product to be displayed to obtain a target image.

In this embodiment, the product image can be detected to obtain the attribute information of the product to be displayed in the product image. In the process of using text features to guide the attention processing model in the image processing model to perform attention processing on the product image, the text features can be used to guide the attention processing model to perform attention processing on the attribute information of the product to be displayed to obtain the target image, wherein the attribute information can be the appearance information and material information of the product, etc. For example, if the product to be displayed is clothing, the attribute information of the clothing can be the color, detailed texture and trademark (logo) of the clothing. Attribute information can also be called knowledge information or knowledge injection information. It should be noted that the above-mentioned attribute information is only for example purposes and is not specifically limited here. It can be determined based on the actual appearance of the product to be displayed.

Optionally, a clothing detection module may be provided in the embodiment of the present application. The clothing detection module may be used to identify and analyze the products to be displayed in the product image and determine the corresponding attribute information. It should be noted that the clothing in the above clothing detection model is only a name restriction, and does not determine the attribute information of the clothing, and other products to be displayed may also be analyzed.

Optionally, the clothing images uploaded by users are passed through a clothing detection module, which is responsible for identifying and cropping out the main clothing area for subsequent processing. This step is to ensure that the subsequent model can focus on the clothing itself without being disturbed by the background or other irrelevant factors. The cropped main clothing area is fed into a VAE model. VAE is a generative model that can learn the potential representation of clothing and provide information such as the color, detailed texture, etc. of clothing. Through the VAE model, the features of clothing can be extracted and encoded for subsequent processing and knowledge injection.

Optionally, the output of the VAE model is fed into a subsequent U-Net model. U-Net is a convolutional neural network used for image segmentation and is commonly used in image processing tasks. Here, U-Net may be used to further process the clothing image, extract more detailed information, and inject knowledge information such as the clothing logo for the final virtual try-on effect.

The entire process mentioned above involves multiple steps such as image detection, feature extraction and knowledge injection, and combines multiple deep learning models to achieve clothing image processing and virtual try-on effect generation. This process plays an important role in virtual try-on technology, providing users with a more realistic and personalized try-on experience.

As an optional implementation, the attention processing model includes: a first self-attention processing model and a first cross-attention processing model, wherein the attention processing model is guided by using text features to perform attention processing on the attribute information of the product to be displayed to obtain a target image, including: using the first self-attention processing model to perform self-attention processing on the attribute information of the product to be displayed to obtain a first self-attention processing result; using the first cross-attention processing model to perform cross-attention processing on the first self-attention processing result and text features to obtain a first cross-attention processing result; generating a target image based on the first cross-attention processing result.

In this embodiment, in the process of using text features to guide the attention processing model to perform attention processing on the attribute information of the product to be displayed to obtain the target image, the first self-attention processing model in the attention processing model can be used to perform self-attention processing on the attribute information of the product to be displayed to obtain a first self-attention processing result. The first cross-attention processing model in the attention processing model can also be used to perform cross-attention processing on the first self-attention processing result and the text features to obtain a first cross-attention processing result. The first cross-attention processing result can be memorized to generate a target image, wherein the attention processing model can include a first self-attention processing model and a first cross-attention processing model. The first self-attention processing model can be a self-attention (Self Attention ion) module. The first cross-attention processing model can be a Cross Attention module.

Optionally, the TryOn model implements knowledge injection through a dual Unet architecture based on the above-mentioned product image and text description information, that is, the clothing image and text description, and finally outputs the clothing upper body image. Among them, the dual Unet architecture generally refers to a neural network structure, which consists of two interrelated Unet networks, which are used to encode and decode the input information respectively. Through this structure, the TryOn model can integrate information from the clothing image, skeleton image and text description to generate the final clothing upper body image. This structure allows the model to process information from different inputs at the same time and inject relevant knowledge, such as posture and clothing details, when generating the clothing upper body image. This method helps to achieve personalized customization for user needs and provide a more accurate and expected virtual try-on effect. Overall, this dual Unet architecture TryOn model combines multiple input information and realizes the generation of clothing upper body images through knowledge injection, thereby providing a more personalized and accurate virtual try-on effect.

In the embodiment of the present application, the architecture of the TryOn model is a dual U-Net, which is used to realize knowledge injection, such as injecting posture, clothing details, etc., and finally outputting a clothing upper body picture. U-Net is a convolutional neural network commonly used for image segmentation. This dual U-Net architecture can effectively integrate the information of clothing pictures and model pictures, and realizes knowledge injection based on the text description entered by the user to generate the final virtual try-on effect. This architecture plays an important role in virtual try-on technology, providing users with a more realistic and personalized try-on experience.

Optionally, the TryOn model obtains the features of the text description through a text encoder and injects these features into the Cross Attention module of the dual U-Net. This design ensures that the text description entered by the user can directly affect the final virtual try-on effect, so that the user's personalized needs and customization requirements can be fully considered.

As an optional implementation, the method also includes: performing posture detection on the carrier to obtain posture information of the carrier, wherein the posture information is used to represent the posture displayed by the carrier in the simulated wearing performance result; using a first self-attention processing model to perform self-attention processing on the attribute information of the product to be displayed to obtain a first self-attention processing result, including: using the first self-attention processing model to perform self-attention processing on the attribute information and posture information of the product to be displayed to obtain a first self-attention processing result.

In this embodiment, the carrier may be subjected to posture detection to obtain posture information of the carrier. In the process of using the first self-attention processing model to perform self-attention processing on the attribute information of the displayed product to obtain the first self-attention processing result, the first self-attention processing model may be used to perform self-attention processing on the attribute information and posture information of the displayed product to obtain the first self-attention processing result, wherein the posture detection may be skeleton detection. The posture information may be a skeleton (pose) diagram obtained by skeleton detection.

Optionally, images of models in any pose uploaded by users can be subjected to skeleton detection.

Optionally, the user uploads a model image in any pose, which can be an actual photo or a simulated image. Through the open source real-time multi-person pose estimation library (Openpose) and other human pose estimation algorithms, the uploaded model image is subjected to skeleton detection to obtain the corresponding skeleton image (pose). This process identifies the key joints and postures of the model's body and generates a skeleton image containing posture information. Among them, Openpose can detect the key points of the human skeleton and estimate the human posture. This library uses deep learning technology, based on convolutional neural networks and computer vision algorithms, and can accurately detect key parts of the human body from images or videos, such as key points such as the head, shoulders, elbows, wrists, hips, knees, ankles, etc. Openpose can also perform multi-person pose estimation, and can accurately segment each person's posture even when there are multiple people overlapping. This makes Openpose have important application value in fields such as human motion analysis, human posture recognition, and virtual fitting.

Optionally, the skeleton image can be used as the second standard input of the Tryon model to control the pose of the model in the virtual try-on model. By combining this skeleton information, the virtual try-on model can ensure that the clothing presents the right pose and posture on the model. The purpose of this step is to ensure that the virtual try-on model can be properly adjusted and displayed according to the pose in the model image, thereby providing a more realistic and practical try-on effect. The whole process combines image processing and posture control technology to provide users with a more personalized and realistic virtual try-on experience.

As an optional implementation, performing posture detection on the carrier to obtain posture information of the carrier includes: calling a posture coding model, performing posture coding on the carrier, and obtaining skeleton features of the carrier, wherein the posture information includes skeleton features.

In this embodiment, in the process of performing posture detection on the carrier to obtain the posture information of the carrier, the posture coding model can be called to perform posture coding on the carrier to obtain the skeleton features of the carrier.

Optionally, the pose information may include skeleton features. The pose encoding model may include a pose encoder, as well as the above-mentioned Openpose. Pose encoder can encode the information of human pose into a vector or feature representation so that the computer can better understand and process the pose information. Pose encoder is commonly used in tasks such as pose estimation, action recognition, and human pose analysis.

Optionally, the received model image can be analyzed by a pose encoder to obtain skeleton encoding features. These encoding features can be used as common inputs together with the original input features of the main U-Net. Such a setting can ensure that the model's posture information can directly affect the final virtual fitting effect, so that the model's posture and movement can be accurately reflected in the generated clothing upper body image.

As an optional implementation, a posture coding model is used to perform posture coding on a carrier to obtain skeleton features of the carrier, including: acquiring an object image, wherein the image content of the object image includes an object in any posture that allows the product to be worn to be displayed; using the posture coding model to identify the object in the object image, and determining the identified object as a carrier, performing posture coding on the carrier to obtain skeleton features of the carrier.

In this embodiment, an object image can be obtained in the process of using a posture coding model to perform posture coding on a carrier to obtain skeleton features. The posture coding model can be used to identify an object in the object image, and the identified object is determined as a carrier, and the carrier is posture-encoded to obtain the skeleton features of the carrier, wherein the image content of the object image may include an object in any posture that allows the product to be displayed to be worn, and the object image may be a model image. The skeleton features can be used to represent the posture of the carrier when displaying the product being worn, and can be the encoding features of the skeleton obtained by the pose encoder.

Optionally, skeleton detection is performed on the model image uploaded by the user, and the detected skeleton image is used as the standard input of the Tryon model to control the pose of the model in the generated image.

Optionally, users can upload pictures of models in any pose, which may be photos or images of models in different actions or poses. Skeleton detection can be performed through OpenPose. Since OpenPose is an open source library for human pose estimation, it can detect the key points of the human skeleton in the image, including the head, shoulders, elbows, wrists, hips, knees, ankles and other key points. Therefore, through OpenPose, skeleton detection can be performed on the model picture uploaded by the user to obtain the corresponding skeleton map. The skeleton map obtained by the above process will be used as one of the inputs of the Tryon model to control the pose of the model in the generated image. Such a setting can ensure that the pose of the model in the generated virtual try-on image is consistent with the model picture uploaded by the user, thereby providing a more realistic and practical try-on effect.

Optionally, through the above method, the Tryon model can generate a virtual try-on effect corresponding to the pose information in the model picture uploaded by the user. This will provide users with a more personalized and realistic try-on experience, and also help to improve the effect and attractiveness of product display.

As an optional implementation, the method also includes: using a first residual network model in the image processing model to perform residual learning on skeleton features; using a first self-attention processing model to perform self-attention processing on attribute information and posture information of the product to be displayed, and obtaining a first self-attention processing result, including: using the first self-attention processing model to perform self-attention processing on the attribute information of the product to be displayed and the learned skeleton features, and obtaining a first self-attention processing result.

In this embodiment, the first residual network model in the image processing model can be used to perform residual learning on the skeleton features. In the process of using the first self-attention processing model to perform self-attention processing on the attribute information and posture information of the displayed product, the first self-attention processing model can be used to perform self-attention processing on the attribute information of the displayed product and the learned skeleton features to obtain the first self-attention processing result, wherein the first residual network model can be a residual network (ResNet) module.

Optionally, in the process of residual learning of the encoded features of the skeleton obtained by the pose encoder using ResNet, a residual block can be defined, including one or more convolutional layers and activation functions. These layers can be used to learn residual information in the skeleton features. The skeleton encoded features obtained by the pose encoder are taken as input, and after being processed by the residual block, the learned residual features are added to the original input features. Such a setting allows the network to learn the difference between the input features and the expected output, thereby improving the performance and training effect of the model. Residual blocks with residual connections can be stacked into a deep network to further learn and extract the encoded features of the skeleton. Such a design can help the network better capture and utilize the input skeleton information and improve the representation ability of the model.

Optionally, through the above process, ResNet can be used to perform residual learning on the encoded features of the skeleton obtained by the pose encoder, thereby improving the model's learning and representation capabilities for skeleton information. This will help improve the performance of the model in pose estimation and related tasks.

As an optional implementation, a product image is detected to obtain attribute information of a product to be displayed in the product image, including: calling an image feature extraction model to extract original image features from the product image; using a second residual network model in the image processing model to perform residual learning on the original image features; using a second self-attention processing model in the image processing model to perform self-attention processing on the learned original image features to obtain a second self-attention processing result, wherein the second self-attention processing result is used to represent image features whose size is greater than a size threshold, and the second self-attention processing result is used to represent the attribute information of the product to be displayed; using a first self-attention processing model to perform self-attention processing on the attribute information of the product to be displayed and the learned skeleton features to obtain a first self-attention processing result, including: using the first self-attention processing model to receive the second self-attention processing result output by the second self-attention processing model, and performing self-attention processing on the second self-attention processing result and the learned skeleton features to obtain the first self-attention processing result.

In this embodiment, in the process of detecting the product image and obtaining the attribute information of the product to be displayed in the product image, the image feature extraction model can be called to extract the original image features from the product image. The second residual network model in the image processing model can be used to perform residual learning on the original image features. The second self-attention processing model in the image processing model can be used to perform self-attention processing on the learned original image features to obtain a second self-attention processing result. The first self-attention processing model can be used to receive the second self-attention processing result output by the second self-attention processing model, and the second self-attention processing result and the skeleton features of the learner can be self-attention processed to obtain the first self-attention processing result.

Optionally, the second residual network model corresponds to the first residual network model and may be ResNet. The image feature extraction module may be an image encoding network module, for example, may be VAE. The original image features may be the original features in the product image. The second self-attention processing model may correspond to the first self-attention processing and may include Self Attention and Cross Attention. The second self-attention processing result may be used to represent image features whose size is greater than a size threshold. The second self-attention processing result may be used to represent attribute information of the product to be displayed.

Optionally, the size threshold may be a preset size, or a size set according to actual conditions, for example, it may be preset to 1024*1024. If the size is greater than the size threshold, it may be indicated that the image feature is a large size. It should be noted that the size and setting method of the above-mentioned size threshold are only for example description and are not specifically limited here.

Optionally, in an embodiment of the present application, for knowledge injection of clothing, a reference U-Net network may be used in the TryOn model to extract large-size (e.g., 1024*1024) clothing image features. In each SelfAttention module, the extracted clothing image features are directly injected into the main U-Net and directly connected in series with the original features. Such a setting can ensure that the detailed features of the clothing image can be effectively integrated into the main U-Net, thereby affecting the final virtual try-on effect, wherein the Unet network extracts large-size (1024*1024) image features, which means that the Unet network can capture higher-resolution image features when processing images. Typically, the Unet network is often used for image segmentation tasks, and its structural design enables detailed and rich feature information to be extracted from large-size input images, wherein the Reference U-Net may also be referred to as ReferenceNet or Reference unet.

Alternatively, for virtual try-on technology, the Unet network extracting large-scale image features may mean that the model can better understand the details, textures, and other important information in the clothing image, making the final virtual try-on effect more realistic and in line with expectations. This also shows that the technology has a strong adaptability to processing high-resolution images and can maintain high details and information richness when processing images. Therefore, the Unet network extracting large-scale image features indicates that the model can better understand and utilize the information in high-resolution images to improve the realism and quality of the virtual try-on effect.

For example, the knowledge injection of clothing relies on a Reference Unet network to extract large-scale image features, which are directly injected into the main U-Net model (main Unet) in each Self Attention module and directly connected in series with the original features.

Optionally, the U-Net model in the embodiment of the present application may include Reference Unet and mainUnet.

As an optional implementation, the method also includes: using a second cross-attention processing model in the image processing model to perform cross-attention processing on the second self-attention processing result and the text feature to obtain a second cross-attention processing result; determining the second cross-attention processing result as the original image feature, determining the next residual network model of the second residual network model in the image processing model as the second residual network model, determining the next self-attention processing model of the second self-attention processing model in the image processing model as the second self-attention processing model, and determining the next cross-attention processing model of the second cross-attention processing model in the image processing model as the second cross-attention processing model, and starting from the following steps until the second residual network model has no next residual network model in the image processing model, the second self-attention processing model has no next self-attention processing model in the image processing model, and the second cross-attention processing model has no next cross-attention processing model in the image processing model: using the second residual network model in the image processing model to perform residual learning on the original image feature.

In this embodiment, the second cross-attention processing model in the image processing model can be used to perform cross-attention processing on the second self-attention processing model and the text feature to obtain a second cross-attention processing result. The second cross-attention processing result can be determined as the original image feature, and the next residual network model of the second residual network model in the image processing model can be determined as the second residual network model. The next self-attention processing model of the second self-attention processing model in the graphic processing model can be determined as the second self-attention processing model. The next cross-attention processing model of the second cross-attention processing model in the image processing model can be determined as the second cross-attention processing model. Until the second residual network model has no next residual network model in the image processing model, the second self-attention processing model has no next self-attention processing model in the image processing model, and the second cross-attention processing model has no next cross-attention processing model in the image processing model, the second residual network model in the image processing model can be used to perform residual learning on the original image feature.

Optionally, a TryOn model of a multi-layer neural network structure is mentioned in an embodiment of the present application, including multi-layer cross-attention processing, self-attention processing, residual network and other modules. The second self-attention processing result and the text feature are subjected to cross-attention processing, that is, the self-attention processing result and the text feature are subjected to cross-attention processing, and the self-attention result in the image processing is associated and integrated with the text feature. The second cross-attention processing result, that is, the result obtained after the cross-attention processing, can be a new feature representation combining the image processing result and the text feature. The second cross-attention processing result is determined as the original image feature, emphasizing that this result will be regarded as the input of the next step. This structure in the embodiment of the present application involves a complex neural network architecture and a variety of deep learning techniques for the fusion and learning of image and text features.

In the embodiment of the present application, the multi-level structure allows the fusion of image and text features at different levels. This helps the TryOn model to better understand and utilize the relationship between different types of features, thereby improving the representation and generalization capabilities of the TryOn model. The multi-level structure allows the abstract representation of features to be learned at different levels. Through layer-by-layer learning and extraction, the TryOn model can gradually understand and utilize more abstract and complex features, thereby improving the representation and understanding capabilities of data. The multi-level structure can focus on and integrate information at different levels at different levels. This helps the TryOn model to understand the input data more comprehensively and analyze and process it from multiple angles. Through the multi-level structure, the TryOn model can better utilize the information in the data, thereby improving the performance and effect of the TryOn model. This structure allows the TryOn model to better utilize the rich information in the data and improve the performance of the TryOn model in complex tasks. The multi-level structure can help the TryOn model better understand the inherent structure and characteristics of the input data, thereby improving the generalization ability of the TryOn model and making it more adaptable to new data.

In summary, the multi-level structure has many benefits and beneficial effects, such as feature fusion, multi-level learning, focusing on information at different levels, improving the performance of the TryOn model and improving the generalization ability, which helps to improve the representation ability and task performance of the TryOn model.

As an optional implementation, the method also includes: determining the first cross-attention processing result as a skeleton feature; determining the next residual network model of the first residual network model in the image processing model as the first residual network model, determining the next self-attention processing model of the first self-attention processing model in the image processing model as the first self-attention processing model, determining the next cross-attention processing model of the first cross-attention processing model in the image processing model as the first cross-attention processing model, and starting from the following steps until the first residual network model has no next residual network model in the image processing model, the first self-attention processing model has no next self-attention processing model in the image processing model, and the first cross-attention processing model has no next cross-attention processing model in the image processing model: using the first residual network model in the image processing model to perform residual learning on the skeleton features, wherein the second self-attention processing model corresponds one-to-one to the first self-attention processing model.

In this embodiment, the first cross-attention processing result can be determined as a skeleton feature. The next residual network model of the first residual network model in the image processing model can be determined as the first residual network model, the next self-attention processing model of the first self-attention processing model in the image processing model can be determined as the first self-attention processing model, and the next cross-attention processing model of the first cross-attention processing model in the image processing model can be determined as the first cross-attention processing model. Until the first residual network model does not have a next residual network model in the image processing model, the first self-attention processing model does not have a next self-attention processing model in the image processing model, and the first cross-attention processing model does not have a next cross-attention processing model in the image processing model, the first residual network model in the image processing model can be used to perform residual learning on the skeleton features, wherein the second self-attention processing model corresponds one-to-one to the first self-attention processing model.

Optionally, the TryOn model in the embodiment of the present application has a complex multi-layer neural network structure. Determining the first cross-attention processing result as a skeleton feature may mean emphasizing that this result will be regarded as the input of the next step. The next residual network model of the first residual network model in the image processing model is determined as the first residual network model, that is, the first residual network model is used as the input of the next step. The first residual network model in the image processing model is used to perform residual learning on the skeleton features, that is, in the process of performing residual learning on the skeleton features, the first residual network model is used to learn and extract features. This complex loop structure can be for deep learning and extraction of skeleton features in images. This multi-layer structure helps the TryOn model to more deeply understand and learn the skeleton information in the image, and improves the TryOn model's ability to understand posture and movement.

In the embodiment of the present application, the TryOn model architecture fully considers the clothing image features, model posture information and text descriptions entered by the user, and through detailed knowledge injection design, ensures that these important information can effectively affect the final virtual try-on effect. Such an architecture is of great significance in virtual try-on technology, providing users with a more realistic and personalized try-on experience.

As an optional implementation, the method further includes: performing image enhancement on the target image; and outputting the enhanced target image.

In this embodiment, the target image may also be enhanced, and the enhanced target image may be output.

Optionally, the target image generated by the image processing model is processed by an image enhancement strategy to further improve the quality and realism of the image, thereby obtaining a final target image, wherein the image enhancement strategy may include adjusting sharpness, contrast, saturation, brightness, etc. It should be noted that the above-mentioned image enhancement strategy is only for illustration and is not specifically limited here.

In the embodiment of the present application, the above-mentioned image enhancement strategy helps to improve the visual effect of the virtual try-on result, so that the virtual try-on image finally presented is clearer and more vivid, and meets the aesthetic requirements of the user. Such processing can effectively enhance the effect of the image and provide a more realistic virtual try-on effect, thereby better meeting the expectations of the user.

Optionally, the clothing image output from the TryOn model is enhanced to obtain the final virtual try-on result. Image enhancement is an image processing technique that can improve the visual quality of an image by adjusting various attributes of the image. These adjustments can include sharpening, contrast enhancement, saturation adjustment, and brightness adjustment.

Optionally, sharpening the image can enhance the details and edges in the image, making the clothing upper body picture clearer and more realistic. Increasing the contrast can make the colors and light and dark areas in the image more distinct, making the clothing upper body picture more vivid and attractive. Adjusting the saturation can increase or decrease the intensity and vividness of the colors in the image to make the colors of the clothing upper body picture more realistic and attractive. Adjusting the brightness can change the overall brightness of the image so that the clothing upper body picture presents a satisfactory effect under different lighting conditions. Through the above-mentioned image enhancement strategy, it can be ensured that the final virtual try-on result achieves a satisfactory visual effect and can better meet the needs and expectations of users. This image enhancement technology can improve the realism and attractiveness of the virtual try-on effect and provide users with a more realistic and personalized try-on experience.

As an optional implementation manner, the text description information includes at least one of the following: display strategy information of the product to be displayed during the display process, background content of the target image to be generated, and attribute information of the carrier.

In this embodiment, the text description information may include at least one of the following: display strategy information of the product to be displayed during the display process, background content of the target image to be generated, and attribute information of the carrier, wherein the display strategy information may be used to indicate the way of displaying the product, for example, if the product to be displayed is clothing, the display strategy information may be whether the clothing is open-necked, rolled-up, or layered. The background content may be the screen background when the product is displayed, for example, a white wall, a landscape, etc. The attribute information may be model attributes, for example, gender, hairstyle, accessories, height, weight, and body shape, etc.

It should be noted that the above-mentioned display strategy information, background content and attribute information are only for illustration and are not specifically limited here. As long as the information can reflect the user's demand for the wearing effect of the displayed product, it is within the protection scope of the embodiments of this application.

Optionally, the display strategy information may include the way of displaying the product, such as whether the clothing is open-necked, with rolled-up sleeves, or layered. This information helps to determine the specific display method of the product, so that the generated target image is more in line with the expected effect and can better meet the personalized needs of users. The background content can describe the background of the picture when the product is displayed. This information helps to determine the background environment when the product is displayed, so that the generated target image has a richer and more realistic visual effect, improving the display effect of the product. The attribute information of the carrier may include the model's gender, hairstyle, accessories, height, weight and body shape. This information helps to determine the appropriate model and carrier, so that the generated target image is more personalized and targeted, and can better show the product effect.

In an embodiment of the present application, the above information can help determine the way to display the product, the background environment and the suitable model, so that the generated target image is more personalized and targeted, and can better show the characteristics and effects of the product. The target image generated according to the information described in the text is more in line with the user's expectations, can improve the user's understanding and satisfaction with the product, and promote consumers' willingness to buy. The information described in this description helps to determine the way to display the product and the background environment, as well as the suitable model, so as to optimize the display effect of the product and improve the attractiveness and competitiveness of the product. In summary, the text description information includes display strategy information, background content and carrier attribute information, which is helpful for personalized customization, improving user satisfaction and optimizing product display effects, and has important beneficial effects on existing inventions.

The embodiment of the present application also provides a method for processing an image from a clothing scene. FIG3 is a flow chart of a method for processing an image according to the embodiment of the present application. As shown in FIG3 , the method may include the following steps:

Step S302, identifying a virtual clothing store deployed on an e-commerce platform, and identifying clothing product images to be processed from the virtual clothing store, wherein the image content of the clothing product images includes at least one clothing product to be displayed.

In the technical solution provided in the above step S302 of the present application, the virtual clothing store on the e-commerce platform can be identified, and the clothing product image to be processed can be identified from the virtual clothing store, wherein the image content of the clothing product image can include the clothing product to be displayed. The e-commerce platform can also be called an e-commerce platform.

Optionally, when a user browses clothing products in a virtual clothing store in an e-commerce platform through an operation interface of a terminal device, if the user needs to determine and display the wearing performance of a certain clothing product on a carrier, the clothing product that needs to be displayed can be determined in the corresponding virtual clothing store. For example, operations such as clicking can be performed on the operation interface to select the clothing product to be displayed from a large number of clothing products.

Step S304, obtaining text description information corresponding to the clothing product to be displayed in the clothing product image, wherein the text description information is used to at least describe the wearing performance result displayed when the clothing product to be displayed is worn by the carrier.

In the technical solution provided in the above step S304 of the present application, after the clothing product image to be processed is identified from the virtual clothing store, text description information corresponding to the clothing product to be displayed in the clothing product image can be obtained.

Optionally, if the user of the e-commerce platform needs to obtain the wearing effect of a certain clothing product, after determining the clothing product material corresponding to the clothing product to be displayed on the operation interface, the user can enter the appearance characteristics of the carrier that can describe the wear and display of the product in the corresponding text input box on the operation interface, or the posture characteristics of the carrier when the product is worn and displayed, and the way the carrier wears the product can also be entered. The text of the above-mentioned appearance characteristics, wearing methods and posture characteristics can be used as the text description information of the product to be displayed.

Step S306, using the text description information to guide the image processing model to analyze the clothing product image to obtain a target image, wherein the image processing model is obtained by training the text image model, and the image content of the target image is used to simulate the wearing performance result.

In the technical solution provided in the above step S306 of the present application, after obtaining the text description information corresponding to the clothing to be displayed in the clothing product image, the text description information can be used to guide the image processing model to analyze the clothing product material to obtain the target image.

Optionally, after obtaining the clothing product image and the corresponding text description information, an image processing model can be used to analyze the text description information and the clothing product image, so as to construct a target image that meets the user's requirements in the text description information for displaying the wearing effect of the clothing product.

For example, after the server receives the clothing product image and text description information, it can call the image processing model deployed in the server's database to analyze the clothing product image and text description information, thereby simulating the wearing performance of the clothing product image when the carrier is worn in accordance with the text description information, and generating a corresponding target image.

Step S308: sending the target image to the e-commerce platform.

In the technical solution provided in the above step S308 of the present application, after using the text description information to guide the image processing model to analyze the clothing product image and obtaining the target image, the target image can be sent to the e-commerce platform.

Optionally, after the target image is determined by the server, the target image can be transmitted to an operation interface on a corresponding user's terminal device through a network. The target image can be displayed on an e-commerce platform on the operation interface.

In an embodiment of the present application, a virtual clothing store deployed on an e-commerce platform is identified, and a clothing product image to be processed is identified from the virtual clothing store, wherein the image content of the clothing product image includes at least one clothing product to be displayed; text description information corresponding to the clothing product to be displayed in the clothing product image is obtained, wherein the text description information is used to at least describe the wearing performance result displayed by the clothing product to be displayed when it is worn by a carrier; using the text description information, an image processing model is guided to analyze the clothing product image to obtain a target image, wherein the image processing model is obtained by training a text image model, and the image content of the target image is used to simulate the wearing performance result; the target image is sent to the e-commerce platform, thereby achieving the technical effect of improving the wearing performance result effect simulated in the image, and solving the technical problem of poor wearing performance result effect simulated in the image.

The embodiment of the present application also provides an image processing method from the human-computer interaction side. FIG4 is a flow chart of an image processing method according to the embodiment of the present application. As shown in FIG4, the method may include the following steps:

Step S402, in response to an image input operation on the operation interface, displaying a product image to be processed on the operation interface, wherein the image content of the product image includes at least one product to be displayed.

In the technical solution provided in the above step S402 of the present application, after the image input operation on the operation interface is detected, the product image to be processed can be displayed on the operation interface.

Optionally, if users or merchants of the e-commerce platform need to obtain the wearing effect of a certain product, they can perform corresponding input operations on the operation interface of the corresponding terminal device. For example, they can upload the product image corresponding to the product to be displayed on the operation interface, or filter out the image of the product that needs to display the wearing effect from the images of a large number of products displayed on the operation interface as the product image. After determining the product image on the operation interface of the terminal device, the product image can be transmitted to the server through the network. The received product image can be identified in the server to determine the product that needs to be displayed for the wearing effect.

It should be noted that the above-mentioned image input operation is only for illustration and no specific limitation is made here.

Step S404, in response to the text input operation on the operation interface, displaying text description information corresponding to the product to be displayed in the product image on the operation interface, wherein the text description information is used to at least describe the wearing performance result displayed by the product to be displayed when it is worn by the carrier.

In the technical solution provided in the above step S404 of the present application, after a text input operation on the operation interface is detected, text description information corresponding to the product to be displayed in the product image can be displayed on the operation interface.

Optionally, if the user of the e-commerce platform needs to obtain the wearing effect of a certain product, then after uploading the product image on the operation interface, the text input operation can be performed on the operation interface. For example, the appearance characteristics of the carrier that can describe the wearable display of the product can be entered in the corresponding text input box on the operation interface, and the posture characteristics of the carrier when the product is worn and displayed can also be entered. The way in which the carrier wears the product can also be entered. The text of the above-mentioned input appearance characteristics, wearing methods and posture characteristics can be used as the text description information of the product to be displayed. It should be noted that the above-mentioned text input operation is only for example and is not specifically limited here.

Step S406, in response to the image generation operation on the operation interface, a target image matching the product image and the text description information is displayed on the operation interface, wherein the target image is obtained by analyzing the product image by guiding the image processing model using the text description information, the image processing model is obtained by training the text image model, and the image content of the target image is used to simulate the wearing performance result.

In the technical solution provided in the above step S406 of the present application, after the image generation operation on the operation interface is detected, a target image matching the product image and text description information can be displayed on the operation interface.

Optionally, after inputting the product image and text description information on the operation interface, an image generation operation can be performed on the operation interface, for example, the "OK" control on the operation interface can be clicked to trigger the server to process the received product image and text description information. It should be noted that the above-mentioned image generation operation is only for illustration and is not specifically limited at this time.

Optionally, after the server receives the product image and text description information, it can call the image processing model deployed in the server's database to analyze the product image and text description information, so as to simulate the wearing performance results of the product to be displayed when the carrier is worn in accordance with the text description information, and generate the corresponding target image.

Optionally, after the server determines the target image, the target image can be transmitted to the terminal device via the network, and the target image can be displayed on the operation interface of the user's terminal device.

In an embodiment of the present application, in response to an image input operation on an operation interface, a product image to be processed is displayed on the operation interface, wherein the image content of the product image includes at least one product to be displayed; in response to a text input operation on the operation interface, text description information corresponding to the product to be displayed in the product image is displayed on the operation interface, wherein the text description information is used to at least describe the wearing performance result displayed by the product to be displayed when it is worn by a carrier; in response to an image generation operation on the operation interface, a target image matching the product image and the text description information is displayed on the operation interface, wherein the target image is obtained by using the text description information to guide the image processing model to analyze the product image, the image processing model is obtained by training the text image model, and the image content of the target image is used to simulate the wearing performance result, thereby achieving the technical effect of improving the effect of the simulated wearing performance result in the image, and solving the technical problem of poor effect of the simulated wearing performance result in the image.

The embodiment of the present application further provides an image processing method. FIG5 is a flow chart of an image processing method according to the embodiment of the present application. As shown in FIG5, the method may include the following steps:

Step S502: displaying a product image to be processed on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the image content of the product image includes at least one product to be displayed.

In the technical solution provided in the above step S502 of the present application, the product collection is displayed on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device.

Optionally, using VR devices or AR devices to display product images can bring more intuitive and vivid display effects. Through VR devices, users can immerse themselves in a virtual environment, present product images in three-dimensional space, and freely observe the appearance and details of the product in 360 degrees, and fully understand the style and characteristics of the product from different angles. AR devices can superimpose product images on real scenes, allowing users to observe the virtual presentation of products directly in reality through AR glasses or mobile phone screens, and observe the size, proportion and adaptability of products in actual environments, providing users with a more realistic experience. Displaying product images through VR and AR devices can allow users to understand the appearance and characteristics of products more intuitively, helping users to better select and make product purchases. At the same time, it can also enhance the attractiveness and fun of product displays, bringing users a new experience.

Step S504, obtaining text description information corresponding to the product to be displayed in the product image, wherein the text description information is used to at least describe the wearing performance result displayed when the product to be displayed is worn by the carrier.

In the technical solution provided in the above step S504 of the present application, text description information corresponding to the product to be displayed in the product image can be obtained.

Optionally, if a customer or merchant of an e-commerce platform needs to obtain the wearing effect of a certain product, after uploading the product image on the operation interface, they can enter text description information that can describe the wearing display of the product in the corresponding text input box on the operation interface.

Step S506, driving the VR device or AR device to use the text description information to guide the image processing model to analyze the product image to obtain a target image, wherein the image processing model is obtained by training the text image model, and the image content of the target image is used to simulate the wearing performance result.

In the technical solution provided in the above step S506 of the present application, the VR device or AR device can be driven to use the text description information to guide the image processing model to analyze the product image and obtain the target image.

Optionally, a VR device or AR device, such as a head-mounted display device or augmented reality glasses, is connected and started. These devices can capture the user's field of view through sensors and cameras and overlay virtual content on the real world. Users can describe the product images they want to analyze by voice input or manually inputting text description information. The VR or AR device can transmit the text description information entered by the user to the image processing model. The image processing model can be a deep learning model for identifying and analyzing images. The image processing model will obtain relevant product images from a database or the Internet based on the text description information entered by the user. The model will then analyze these images and identify target images that match the user's description information. Once the image processing model completes the analysis of the product image, it will return one or more target images that match the user's description information. These target images can be displayed to the user through the VR or AR device so that the user can view and confirm whether they meet their requirements. In short, using text description information to drive the VR or AR device to guide the image processing model to analyze the product image can help users quickly obtain the desired target images, thereby improving work efficiency and user experience.

Step S508: Display the target image on the presentation screen of the VR device or the AR device.

In the technical solution provided in the above step S508 of the present application, the target image can be displayed on a VR device or an AR device.

In an embodiment of the present application, a product image to be processed is displayed on a presentation screen of a virtual reality VR device or an augmented reality AR device, wherein the image content of the product image includes at least one product to be displayed; text description information corresponding to the product to be displayed in the product image is obtained, wherein the text description information is used to at least describe the wearing performance result displayed by the product to be displayed when it is worn by a carrier; the VR device or AR device is driven to use the text description information to guide the image processing model to analyze the product image to obtain a target image, wherein the image processing model is obtained by training a text image model, and the image content of the target image is used to simulate the wearing performance result; the target image is displayed on the presentation screen of the VR device or the AR device, thereby achieving the technical effect of improving the effect of the simulated wearing performance result in the image, and solving the technical problem of poor effect of the simulated wearing performance result in the image.

Example 2

According to an embodiment of the present application, an embodiment of an image processing system is also provided. FIG. 6 is a schematic diagram of an image processing system according to an embodiment of the present application. As shown in FIG. 6 , the image processing system 600 may include: a client 601 and a server 602.

Client 601 is used to upload a product image to be processed, wherein the image content of the product image includes at least one product to be displayed; and upload text description information corresponding to the product to be displayed in the product image, wherein the text description information is used to at least describe the wearing performance result displayed by the product to be displayed when it is worn by the carrier.

In this embodiment, the product image to be processed can be uploaded through the client 601. The text description information corresponding to the product to be displayed in the product image can also be uploaded on the client 601.

Optionally, if a user or merchant of an e-commerce platform needs to obtain the wearing effect of a certain product, they can upload a product image corresponding to the product to be displayed on the operation interface of the corresponding client 601, or filter out the image of the product to be displayed from the images of a large number of products displayed on the operation interface as the product image. After the product image is determined on the operation interface of the terminal device, the product image can be transmitted to the server 602 through the network.

Optionally, if a user of the e-commerce platform needs to obtain the wearing effect of a certain product, after uploading the product image on the operation interface, the user can enter text description information that can describe the wearing display of the product in the corresponding text input box on the operation interface. The text description information can be transmitted to the server 602 via the network.

Server 602 is used to use text description information to guide the image processing model to analyze the product image to obtain a target image, wherein the image processing model is obtained by training the text image model, and the image content of the target image is used to simulate the wearing performance result; and the target image is sent to the client.

In this embodiment, the server 602 may use the text description information to guide the image processing model to analyze the product image to obtain the target image.

Optionally, after the server 602 receives the text description information and product image from the client 601, an image processing model can be used to analyze the text description information and the product image, thereby constructing a target image that meets the user's requirements for the wearing effect of the displayed product in the text description information.

Optionally, after the target image is determined, the target image may be transmitted to the client 601 through a network, and the target image may be displayed on the client 601 .

In an embodiment of the present application, an image processing system 600 is provided. A product image to be processed is uploaded through a client 601, wherein the image content of the product image includes at least one product to be displayed; text description information corresponding to the product to be displayed in the product image is uploaded, wherein the text description information is used to at least describe the wearing performance result displayed when the product to be displayed is worn by a carrier; the server 602 uses the text description information to guide the image processing model to analyze the product image to obtain a target image, wherein the image processing model is obtained by training the Wensheng graph model, and the image content of the target image is used to simulate the wearing performance result; the target image is sent to the client, thereby achieving the technical effect of improving the effect of the simulated wearing performance result in the image, and solving the technical problem of poor effect of the simulated wearing performance result in the image.

It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application, for example, the data for verification, are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

Example 3

Currently, virtual fitting technology can reduce the cost of launching new products by merchants because it can reduce the need for actual fitting and photo shooting, thereby improving the efficiency of launching new products. Virtual fitting technology uses computer vision and image processing technology to merge product images with model graphics to simulate actual wearing effects. Related virtual fitting technology usually requires specific model images, which may pose a challenge to some merchants.

Traditionally, merchants need to spend a lot of resources on model try-on photos before putting new clothing on the shelves, including model fees, shooting costs, and planning of styling, background, and lighting. The emergence of virtual fitting technology provides a solution to this problem. By combining advanced computer vision and image processing technology, virtual fitting technology can merge product images with model graphics to simulate the effect of actual wearing. Merchants can use this technology to quickly generate a large number of high-quality photos of models trying on new products, and select high-quality samples from them, greatly saving time and cost and improving work efficiency. This shows that virtual fitting technology has significant commercial and practical value in the clothing industry.

However, related virtual fitting technologies usually require a specific model picture, which may pose a challenge to some merchants. This may limit the flexibility of merchants when trying on clothes, especially for those who want to provide a personalized fitting experience. The arbitrary use of existing model photos may touch on copyright issues, which may limit the freedom of merchants to use specific model pictures. Restricted by the model's original body shape and the clothes they wear, this may affect the fitting effect, which makes the application of virtual fitting technology subject to certain restrictions. In general, there is still room for further development of virtual fitting technology in overcoming these challenges, such as developing more flexible and freer virtual fitting technology to better meet the needs of merchants and consumers. Therefore, there is still a technical problem that the simulated wearing performance results in the image are poor.

Furthermore, the present application provides a virtual fitting process method based on the Wenshengtu framework, which aims to fundamentally solve the challenges currently faced by virtual fitting technology. This method not only ensures a high degree of realism in simulated fitting, but also allows merchants to construct the model fitting effect only through simple text descriptions, greatly improving the convenience of operation. It also supports the use of semantic descriptions to personalize features such as the background of the picture, the gender and body shape of the model. This means that merchants can customize more easily, and the method avoids copyright issues, greatly improving the service experience of merchants. The innovation and flexibility of this method will bring new development and application prospects to virtual fitting technology. The emergence of the above-mentioned virtual fitting process method fills some gaps in virtual fitting technology, and provides merchants with a more flexible, convenient and personalized virtual fitting experience, effectively solving the challenges faced by existing virtual fitting technology. Thereby achieving the technical effect of improving the effect of the simulated wearing performance results in the image, and solving the technical problem of poor effect of the simulated wearing performance results in the image.

The above method of this embodiment is further introduced below.

In an embodiment of the present application, Figure 7 is a schematic diagram of a virtual try-on process system based on a Wensheng graph framework according to an embodiment of the present application. As shown in Figure 7, the system may include a product image 701, a variational autoencoder 702, that is, a VAE 702, a residual network module 703, a self-attention module 704, a cross-attention module 705, a residual network module 706, a self-attention module 707, a cross-attention module 708, text description information 709, a text encoder 710, original image features 711, a model image 712, a posture encoder 713, a residual network module 714, a self-attention module 715, a cross-attention module 716, a residual network module 717, a self-attention module 718, a cross-attention module 719 and a target image 720.

Optionally, as shown in FIG7 , a dual Unet architecture can be used, which is usually used for virtual try-on models. This architecture allows the model to process two tasks at the same time: one for extracting feature representations of images, and the other for generating the final virtual try-on effect image based on these feature representations. This structure helps the model better understand clothing images and generate virtual try-on effects that meet user expectations. Therefore, the dual Unet architecture is usually used to process virtual try-on tasks, helping the model to better understand the features and details of clothing images, thereby generating realistic virtual try-on effects.

Optionally, as shown in FIG7 , the product image uploaded by the user passes through a clothing detection module, and the main clothing area is cropped out as the first standard input of the Tryon model, and is sequentially sent to the VAE 702 and the subsequent unet model to provide knowledge injection information such as clothing color, detailed texture, and logo.

Optionally, as shown in FIG7 , the user uploads a model image 712 in any pose, performs skeleton detection through openpose, and obtains a corresponding skeleton image as the second standard input of the Tryon model, which is used to control the model pose in the raw image.

Optionally, as shown in FIG. 7 , the user inputs text description information 709, which may include but is not limited to customized model attributes such as the background for specifying the generated image, the model's gender, hairstyle, accessories, height, weight, body shape, or customized clothing attributes such as whether the clothing is open-necked, with rolled-up sleeves, or layered, as the third standard input of the Tryon model.

Optionally, as shown in FIG. 7 , the Tryon model implements knowledge injection (posture, clothing details) through a dual-unet architecture based on the above three standard inputs, and finally outputs a clothing upper body image, that is, a target image 720 .

Optionally, as shown in FIG7 , the knowledge injection of posture obtains the encoded features of the skeleton through the posture encoder 713, and takes them as common input together with the original input features of the main unet. The injection of text description information 709 obtains text features through a text encoder 710, and injects them into the Cross Attention module of the dual unet respectively. The output target image 720 is subjected to the image enhancement strategy (adjusting sharpness, contrast, saturation, brightness), and finally the virtual fitting result is obtained.

Optionally, FIG8 is a schematic diagram of a knowledge injection position according to an embodiment of the present application. As shown in FIG8, ReferenceNet may include a residual network module 801, a self-attention module 802, a cross-attention module 803, and a self-attention module 804. The knowledge injection of clothing relies on a Reference unet network to extract large-size (1024*1024) image features. The self-attention model 804 can transfer its processed results, such as Key and Value, to the self-attention module 802, so that the results of the two self-attention models can be combined to determine the final target image.

Optionally, Figure 9 is a schematic diagram of a knowledge injection method according to an embodiment of the present application. As shown in Figure 9, each Self Attent ion module is directly injected into the main unet and directly connected in series with the original features. That is, each Self Attent ion module in the reference network can be directly injected into the main U-Net model. For example, the Value of ReferenceNet and the Value in the main unet can be connected in series, and the Keys of the two can also be connected in series.

In the embodiments of the present application, the above method cleverly overcomes the limitations of related virtual fitting technologies and provides an innovative virtual fitting method that does not rely on specific model images. Through the text-to-image image generation algorithm, it is allowed to completely freely control the generation of renderings of the model's upper body through text descriptions, without having to deal with complex mask image production or removal of the model's original clothing. While improving the flexibility of clothing, it also eliminates copyright concerns. Through the above method, users can enjoy a seamless, customized and efficient virtual fitting experience, bringing personalization options to a whole new level.

In this embodiment, there is no need to rely on specific model images, overcoming the limitation of traditional methods that require taking photos based on actual models. Users can control the generated model's upper body effect through text descriptions, which provides higher degrees of freedom and customization options while avoiding possible copyright issues, ensuring that merchants can use it with peace of mind. The traditional complex mask image generation strategy required to remove the model's original clothing is omitted in the above method, simplifying the image processing process and being able to flexibly respond to the needs of wearing various styles of clothing.

In an embodiment of the present application, if it is necessary to determine and display the wearing performance of a certain product on a carrier, the product image to be processed of the product can be identified. Obtain text description information for displaying the product on the carrier. The text description information can be used to guide the image processing model pre-trained by the Wensheng graph model to analyze the product image and obtain a target image that can simulate the wearing performance of the product on the carrier. Since the embodiment of the present application takes into account the problems in the relevant technology, a virtual fitting technology using the Wensheng graph framework is proposed. The virtual fitting effect of the product is expressed in words through a simple text description, and the image processing model trained by the Wensheng graph model is used to analyze it, so that a highly realistic model fitting effect image can be generated, thereby eliminating the dependence on specific model pictures, providing users with a product display effect with greater freedom, flexibility and customized selection, and achieving the purpose of reducing the limitations of simulating the wearing performance results of the product through images, thereby achieving the technical effect of improving the effect of the simulated wearing performance results in the image, and solving the technical problem of poor effect of the simulated wearing performance results in the image.

Figure 10 is a schematic diagram of image processing by a computer device according to an embodiment of the present invention. As shown in Figure 10, a request initiated by a user on a client device to simulate the wearing effect of a certain product, that is, an image processing request, can be obtained by calling the first interface. The computer device obtains the product image and text description information received on the client device from the network; based on the product image and text description information received on the client device, the text description information is used to call and guide the image processing model to analyze the product image to obtain the target image.

In an embodiment of the present invention, a product image and text description information received on a client device are obtained by calling a first interface; based on the product image and text description information received on the client device, the text description information is used to call and guide an image processing model to analyze the product image to obtain a target image, thereby achieving a technical effect of improving the effect of a simulated wearing performance result in the image and solving the technical problem of a poor effect of a simulated wearing performance result in the image.

Example 4

According to an embodiment of the present application, there is also provided an image processing device for implementing the image processing method shown in FIG. 2 .

FIG11 is a schematic diagram of an image processing device according to an embodiment of the present application. As shown in FIG11 , the image processing device 1100 may include: a first recognition unit 1102 , a first acquisition unit 1104 , and a first guiding unit 1106 .

The first recognition unit 1102 is configured to recognize a product image to be processed, wherein the image content of the product image includes at least one product to be displayed.

The first acquisition unit 1104 is used to acquire text description information corresponding to the product to be displayed in the product image, wherein the text description information is used to at least describe the wearing performance result displayed when the product to be displayed is worn by the carrier.

The first guiding unit 1106 is used to use the text description information to guide the image processing model to analyze the product image to obtain a target image, wherein the image processing model is obtained by training the text image model, and the image content of the target image is used to simulate the wearing performance result.

Here, the first identification unit 1102, the first acquisition unit 1104 and the first guidance unit 1106 correspond to steps S202 to S206 in Example 1. The three units and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned units can be hardware components or software components stored in a memory (e.g., memory 1504) and processed by one or more processors (e.g., processors 1502a, 1502b..., 1502n), and the above-mentioned units can also be run in the computer terminal A provided in Example 5 as part of the device.

According to an embodiment of the present application, a clothing image processing device for implementing the clothing image processing method shown in FIG. 3 is also provided.

Figure 12 is a schematic diagram of a clothing image processing device according to an embodiment of the present application. As shown in Figure 12, the clothing image processing device 1200 may include: a second recognition unit 1202, a second acquisition unit 1204, a second guiding unit 1206 and a sending unit 1208.

The second identification unit 1202 is used to identify a virtual clothing store deployed on the e-commerce platform, and identify clothing product images to be processed from the virtual clothing store, wherein the image content of the clothing product image includes at least one clothing product to be displayed.

The second acquisition unit 1204 is used to acquire text description information corresponding to the clothing product to be displayed in the clothing product image, wherein the text description information is used to at least describe the wearing performance result displayed when the clothing product to be displayed is worn by the carrier.

The second guiding unit 1206 is used to use the text description information to guide the image processing model to analyze the clothing product image to obtain a target image, wherein the image processing model is obtained by training the text image model, and the image content of the target image is used to simulate the wearing performance result.

The sending unit 1208 is used to send the target image to the e-commerce platform.

It should be noted that the second identification unit 1202, the second acquisition unit 1204, the second guidance unit 1206 and the issuing unit 1208 correspond to steps S302 to S308 in Example 1, and the four units and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned units can be hardware components or software components stored in a memory (e.g., memory 1504) and processed by one or more processors (e.g., processors 1502a, 1502b..., 1502n), and the above-mentioned units can also be run in the computer terminal A provided in Example 5 as part of the device.

According to an embodiment of the present application, there is also provided an image processing device for implementing the image processing method shown in FIG. 4 above.

FIG13 is a schematic diagram of another image processing device according to an embodiment of the present application. As shown in FIG13 , the image processing device 1300 may include: a first display unit 1302 , a second display unit 1304 , and a third display unit 1306 .

The first display unit 1302 is used to respond to the image input operation on the operation interface and display the product image to be processed on the operation interface, wherein the image content of the product image includes at least one product to be displayed.

The second display unit 1304 is used to respond to a text input operation on the operation interface, and display text description information corresponding to the product to be displayed in the product image on the operation interface, wherein the text description information is used to at least describe the wearing performance results displayed by the product to be displayed when it is worn by the carrier.

The third display unit 1306 is used to respond to the image generation operation on the operation interface, and display a target image matching the product image and the text description information on the operation interface, wherein the target image is obtained by analyzing the product image by guiding the image processing model using the text description information, the image processing model is obtained by training the text image model, and the image content of the target image is used to simulate the wearing performance result.

Here, the first display unit 1302, the second display unit 1304, and the third display unit 1306 correspond to steps S402 to S408 in Embodiment 1, and the four units and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in Embodiment 1. It should be noted that the above units may be hardware components or software components stored in a memory (e.g., memory 1504) and processed by one or more processors (e.g., processors 1502a, 1502b..., 1502n), and the above units may also be part of the device and run in the computer terminal A provided in Embodiment 5.

According to an embodiment of the present application, there is also provided an image processing device for implementing the image processing method shown in FIG. 5 .

FIG14 is a schematic diagram of another image processing device according to an embodiment of the present application. As shown in FIG14 , the image processing device 1400 may include: a first display unit 1402 , a third acquisition unit 1404 , a third guiding unit 1406 and a second display unit 1408 .

The first display unit 1402 is used to display the product image to be processed on the presentation screen of the virtual reality VR device or the augmented reality AR device, wherein the image content of the product image includes at least one product to be displayed.

The third acquisition unit 1404 is used to acquire text description information corresponding to the product to be displayed in the product image, wherein the text description information is used to at least describe the wearing performance result displayed when the product to be displayed is worn by the carrier.

The third guiding unit 1406 is used to drive the VR device or AR device to use the text description information to guide the image processing model to analyze the product image to obtain a target image, wherein the image processing model is obtained by training the text image model, and the image content of the target image is used to simulate the wearing performance result.

The second display unit 1408 is used to display the target image on the presentation screen of the VR device or the AR device.

Here, the first display unit 1402, the third acquisition unit 1404, the third guide unit 1406, and the second display unit 1408 correspond to steps S502 to S508 in Example 1. The four units and the corresponding steps have the same examples and application scenarios, but are not limited to the contents disclosed in the above-mentioned Example 1. It should be noted that the above-mentioned units can be hardware components or software components stored in a memory (e.g., memory 1504) and processed by one or more processors (e.g., processors 1502a, 1502b..., 1502n), and the above-mentioned units can also be run in the computer terminal A provided in Example 5 as part of the device.

In the image processing device, if it is necessary to determine and display the wearing performance of a certain product on a carrier, the product image to be processed of the product can be identified. The text description information of the product displayed on the carrier is obtained. The text description information can be used to guide the image processing model pre-trained by the Wensheng graph model to analyze the product image and obtain a target image that can simulate the wearing performance of the product on the carrier. Since the embodiment of the present application takes into account the problems in the related art, a virtual fitting technology using the Wensheng graph framework is proposed. The virtual fitting effect of the product is expressed in words through a simple text description, and the image processing model trained by the Wensheng graph model is used to analyze it, so that a highly realistic model fitting effect image can be generated, thereby eliminating the dependence on specific model pictures, providing users with a product display effect with greater freedom, flexibility and customized selection, and achieving the purpose of reducing the limitations of the wearing performance results of the product simulated by the image, thereby achieving the technical effect of improving the effect of the simulated wearing performance results in the image, and solving the technical problem of poor effect of the simulated wearing performance results in the image.

Example 5

The embodiment of the present application may provide a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal.

Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the image processing method: identifying a product image to be processed, wherein the image content of the product image includes at least one product to be displayed; obtaining text description information corresponding to the product to be displayed in the product image, wherein the text description information is used to at least describe the wearing performance result displayed by the product to be displayed when it is worn by the carrier; using the text description information, guiding the image processing model to analyze the product image to obtain a target image, wherein the image processing model is obtained by training the text image model, and the image content of the target image is used to simulate the wearing performance result.

Optionally, Figure 15 is a block diagram of a computer terminal according to an embodiment of the present application. As shown in Figure 15, the computer terminal A may include: one or more (only one is shown in the figure) processors 1502, a memory 1504 and a transmission device 1506.

Among them, the memory can be used to store software programs and modules, such as the program instructions/modules corresponding to the image processing method and device in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned image processing method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

Optionally, the processor may also execute program codes of the following steps: using an image processing model to extract text features from text description information; using the text features to guide the image processing model to analyze the product image to obtain a target image.

Optionally, the processor may also execute the following program code: using the text feature extraction model in the image processing model to extract text features from the text description information; using the text features to guide the attention processing model in the image processing model to perform attention processing on the product image to obtain the target image.

Optionally, the processor may also execute the following program code: detecting the product image to obtain the attribute information of the product to be displayed in the product image; using text features to guide the attention processing model to perform attention processing on the attribute information of the product to be displayed to obtain the target image.

Optionally, the processor may also execute the program code of the following steps: using a first self-attention processing model to perform self-attention processing on the attribute information of the product to be displayed to obtain a first self-attention processing result; using a first cross-attention processing model to perform cross-attention processing on the first self-attention processing result and text features to obtain a first cross-attention processing result; and generating a target image based on the first cross-attention processing result.

Optionally, the processor may also execute the program code of the following steps: performing posture detection on the carrier to obtain posture information of the carrier, wherein the posture information is used to represent the posture displayed by the carrier in the simulated wearing performance result; performing self-attention processing on the attribute information and posture information of the displayed product using the first self-attention processing model to obtain a first self-attention processing result.

Optionally, the processor may also execute program codes of the following steps: calling a posture coding model, performing posture coding on the carrier, and obtaining skeleton features of the carrier, wherein the posture information includes the skeleton features.

Optionally, the processor may also execute program code of the following steps: obtaining an object image, wherein the image content of the object image includes an object in any posture that allows the product to be worn; identifying the object in the object image using a posture coding model, and determining the identified object as a carrier, performing posture coding on the carrier, and obtaining skeleton features of the carrier.

Optionally, the processor may also execute the program code of the following steps: using the first residual network model in the image processing model to perform residual learning on the skeleton features; using the first self-attention processing model to perform self-attention processing on the attribute information of the displayed product and the learned skeleton features to obtain a first self-attention processing result.

Optionally, the processor may also execute the following program code: calling the image feature extraction model to extract original image features from the product image; performing residual learning on the original image features using the second residual network model in the image processing model; performing self-attention processing on the learned original image features using the second self-attention processing model in the image processing model to obtain a second self-attention processing result, wherein the second self-attention processing result is used to represent image features whose size is greater than a size threshold, and the second self-attention processing result is used to represent attribute information of the product to be displayed; using the first self-attention processing model, receiving the second self-attention processing result output by the second self-attention processing model, and performing self-attention processing on the second self-attention processing result and the learned skeleton features to obtain the first self-attention processing result.

Optionally, the processor may also execute the following program code: using the second cross-attention processing model in the image processing model to perform cross-attention processing on the second self-attention processing result and the text feature to obtain a second cross-attention processing result; determining the second cross-attention processing result as the original image feature, determining the next residual network model of the second residual network model in the image processing model as the second residual network model, determining the next self-attention processing model of the second self-attention processing model in the image processing model as the second self-attention processing model, determining the next cross-attention processing model of the second cross-attention processing model in the image processing model as the second cross-attention processing model, and starting from the following steps until the second residual network model has no next residual network model in the image processing model, the second self-attention processing model has no next self-attention processing model in the image processing model, and the second cross-attention processing model has no next cross-attention processing model in the image processing model: using the second residual network model in the image processing model to perform residual learning on the original image feature.

Optionally, the processor may also execute the following program code: determining the first cross-attention processing result as a skeleton feature; determining the next residual network model of the first residual network model in the image processing model as the first residual network model; determining the next self-attention processing model of the first self-attention processing model in the image processing model as the first self-attention processing model; determining the next cross-attention processing model of the first cross-attention processing model in the image processing model as the first cross-attention processing model; and starting from the following steps until the first residual network model has no next residual network model in the image processing model, the first self-attention processing model has no next self-attention processing model in the image processing model, and the first cross-attention processing model has no next cross-attention processing model in the image processing model: using the first residual network model in the image processing model to perform residual learning on the skeleton features, wherein the second self-attention processing model corresponds one-to-one to the first self-attention processing model.

Optionally, the processor may also execute program codes of the following steps: performing image enhancement on the target image; and outputting the enhanced target image.

Optionally, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: identify a virtual clothing store deployed on the e-commerce platform, and identify a clothing product image to be processed from the virtual clothing store, wherein the image content of the clothing product image includes at least one clothing product to be displayed; obtain text description information corresponding to the clothing product to be displayed in the clothing product image, wherein the text description information is used to at least describe the wearing performance results displayed by the clothing product to be displayed when it is worn by the carrier; use the text description information to guide the image processing model to analyze the clothing product image to obtain a target image, wherein the image processing model is obtained by training the text image model, and the image content of the target image is used to simulate the wearing performance results; send the target image to the e-commerce platform.

Optionally, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: in response to the image input operation on the operation interface, display the product image to be processed on the operation interface, wherein the image content of the product image includes at least one product to be displayed; in response to the text input operation on the operation interface, display the text description information corresponding to the product to be displayed in the product image on the operation interface, wherein the text description information is used to at least describe the wearing performance result displayed by the product to be displayed when it is worn by the carrier; in response to the image generation operation on the operation interface, display the target image matching the product image and the text description information on the operation interface, wherein the target image is obtained by using the text description information to guide the image processing model to analyze the product image, the image processing model is obtained by training the text image model, and the image content of the target image is used to simulate the wearing performance result.

Optionally, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: display the product image to be processed on the presentation screen of the virtual reality VR device or the augmented reality AR device, wherein the image content of the product image includes at least one product to be displayed; obtain text description information corresponding to the product to be displayed in the product image, wherein the text description information is used to at least describe the wearing performance result displayed by the product to be displayed when it is worn by the carrier; drive the VR device or AR device to use the text description information to guide the image processing model to analyze the product image to obtain a target image, wherein the image processing model is obtained by training the text image model, and the image content of the target image is used to simulate the wearing performance result; display the target image on the presentation screen of the VR device or the AR device.

According to an embodiment of the present application, an image processing method is provided. In the embodiment of the present application, if it is necessary to determine and display the wearing performance of a certain product on a carrier, the product image to be processed of the product can be identified. The text description information of the product displayed on the carrier is obtained. The text description information can be used to guide the image processing model pre-trained by the Wensheng graph model to analyze the product image and obtain a target image that can simulate the wearing performance of the product on the carrier. Since the embodiment of the present application takes into account the problems in the related art, a virtual fitting technology using the Wensheng graph framework is proposed. The virtual fitting effect of the product is expressed in words through a simple text description, and the image processing model trained by the Wensheng graph model is used to analyze it, so that a highly realistic model fitting effect image can be generated, thereby eliminating the dependence on specific model pictures, providing users with a product display effect with greater freedom, flexibility and customized selection, and achieving the purpose of reducing the limitations of the wearing performance results of the product simulated by the image, thereby achieving the technical effect of improving the effect of the simulated wearing performance result in the image, and solving the technical problem of poor effect of the simulated wearing performance result in the image.

Those skilled in the art will appreciate that the structure shown in FIG. 15 is for illustration only, and the computer terminal A may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID for short), a PAD, and other terminal devices. FIG. 15 does not limit the structure of the computer terminal A. For example, the computer terminal A may also include more or fewer components (such as a network interface, a display device, etc.) than those shown in FIG. 15 , or may have a configuration different from that shown in FIG. 15 .

A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

Example 6

The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the image processing method provided in the first embodiment.

Optionally, in this embodiment, the computer-readable storage medium may be located in any one of the computer terminals in a computer terminal group in a computer network, or in any one of the mobile terminals in a mobile terminal group.

Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: identifying a product image to be processed, wherein the image content of the product image includes at least one product to be displayed; obtaining text description information corresponding to the product to be displayed in the product image, wherein the text description information is used to at least describe a wearing performance result displayed by the product to be displayed when it is worn by a carrier; using the text description information, guiding an image processing model to analyze the product image to obtain a target image, wherein the image processing model is obtained by training a text image model, and the image content of the target image is used to simulate the wearing performance result.

Optionally, the computer-readable storage medium may also execute program codes for the following steps: extracting text features from text description information using an image processing model; and guiding the image processing model to analyze the product image using the text features to obtain a target image.

Optionally, the computer-readable storage medium may also execute program codes of the following steps: utilizing a text feature extraction model in an image processing model to extract text features from text description information; utilizing text features to guide an attention processing model in an image processing model to perform attention processing on a product image to obtain a target image.

Optionally, the computer-readable storage medium may also execute program codes of the following steps: detecting the product image to obtain attribute information of the product to be displayed in the product image; using text features to guide the attention processing model to perform attention processing on the attribute information of the product to be displayed to obtain the target image.

Optionally, the computer-readable storage medium can also execute the program code of the following steps: using a first self-attention processing model to perform self-attention processing on the attribute information of the product to be displayed to obtain a first self-attention processing result; using a first cross-attention processing model to perform cross-attention processing on the first self-attention processing result and text features to obtain a first cross-attention processing result; based on the first cross-attention processing result, generating a target image.

Optionally, the computer-readable storage medium may also execute program codes of the following steps: performing posture detection on the carrier to obtain posture information of the carrier, wherein the posture information is used to represent the posture displayed by the carrier in the simulated wearing performance result; performing self-attention processing on the attribute information and posture information of the displayed product using a first self-attention processing model to obtain a first self-attention processing result.

Optionally, the computer-readable storage medium may also execute program codes of the following steps: calling a posture coding model, performing posture coding on the carrier, and obtaining skeleton features of the carrier, wherein the posture information includes skeleton features.

Optionally, the computer-readable storage medium may also execute program code for the following steps: obtaining an object image, wherein the image content of the object image includes an object in any posture that allows the product to be worn; identifying the object in the object image using a posture coding model, determining the identified object as a carrier, performing posture coding on the carrier, and obtaining skeleton features of the carrier.

Optionally, the computer-readable storage medium can also execute program codes of the following steps: using the first residual network model in the image processing model to perform residual learning on the skeleton features; using the first self-attention processing model to perform self-attention processing on the attribute information of the displayed product and the learned skeleton features to obtain a first self-attention processing result.

Optionally, the computer-readable storage medium can also execute the following steps of program code: calling the image feature extraction model to extract original image features from the product image; using the second residual network model in the image processing model to perform residual learning on the original image features; using the second self-attention processing model in the image processing model to perform self-attention processing on the learned original image features to obtain a second self-attention processing result, wherein the second self-attention processing result is used to represent image features whose size is greater than a size threshold, and the second self-attention processing result is used to represent attribute information of the product to be displayed; using the first self-attention processing model, receiving the second self-attention processing result output by the second self-attention processing model, and performing self-attention processing on the second self-attention processing result and the learned skeleton features to obtain the first self-attention processing result.

Optionally, the computer-readable storage medium can also execute the following program code: using the second cross-attention processing model in the image processing model to perform cross-attention processing on the second self-attention processing result and the text feature to obtain a second cross-attention processing result; determining the second cross-attention processing result as the original image feature, determining the next residual network model of the second residual network model in the image processing model as the second residual network model, determining the next self-attention processing model of the second self-attention processing model in the image processing model as the second self-attention processing model, and determining the next cross-attention processing model of the second cross-attention processing model in the image processing model as the second cross-attention processing model, and starting from the following steps until the second residual network model has no next residual network model in the image processing model, the second self-attention processing model has no next self-attention processing model in the image processing model, and the second cross-attention processing model has no next cross-attention processing model in the image processing model: using the second residual network model in the image processing model to perform residual learning on the original image feature.

Optionally, the computer-readable storage medium can also execute the program code of the following steps: determining the first cross-attention processing result as a skeleton feature; determining the next residual network model of the first residual network model in the image processing model as the first residual network model, determining the next self-attention processing model of the first self-attention processing model in the image processing model as the first self-attention processing model, determining the next cross-attention processing model of the first cross-attention processing model in the image processing model as the first cross-attention processing model, and starting from the following steps until the first residual network model has no next residual network model in the image processing model, the first self-attention processing model has no next self-attention processing model in the image processing model, and the first cross-attention processing model has no next cross-attention processing model in the image processing model: using the first residual network model in the image processing model to perform residual learning on the skeleton features, wherein the second self-attention processing model corresponds one-to-one to the first self-attention processing model.

Optionally, the computer-readable storage medium may also execute program codes for the following steps: performing image enhancement on the target image; and outputting the enhanced target image.

Optionally, the computer-readable storage medium may also execute program codes of the following steps: identifying a virtual clothing store deployed on an e-commerce platform, and identifying clothing product images to be processed from the virtual clothing store, wherein the image content of the clothing product images includes at least one clothing product to be displayed; obtaining text description information corresponding to the clothing product to be displayed in the clothing product image, wherein the text description information is used to at least describe the wearing performance results displayed by the clothing product to be displayed when it is worn by a carrier; using the text description information, guiding an image processing model to analyze the clothing product image to obtain a target image, wherein the image processing model is obtained by training a text image model, and the image content of the target image is used to simulate the wearing performance results; and sending the target image to the e-commerce platform.

Optionally, the computer-readable storage medium may also execute program codes of the following steps: in response to an image input operation on an operation interface, displaying a product image to be processed on the operation interface, wherein the image content of the product image includes at least one product to be displayed; in response to a text input operation on the operation interface, displaying text description information corresponding to the product to be displayed in the product image on the operation interface, wherein the text description information is used to at least describe the wearing performance result displayed by the product to be displayed when it is worn by the carrier; in response to an image generation operation on the operation interface, displaying a target image matching the product image and the text description information on the operation interface, wherein the target image is obtained by using the text description information to guide the image processing model to analyze the product image, the image processing model is obtained by training the text image model, and the image content of the target image is used to simulate the wearing performance result.

Optionally, the computer-readable storage medium may also execute program code for the following steps: displaying a product image to be processed on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, wherein the image content of the product image includes at least one product to be displayed; obtaining text description information corresponding to the product to be displayed in the product image, wherein the text description information is used to at least describe a wearing performance result displayed by the product to be displayed when the product is worn by a carrier; driving the VR device or AR device to use the text description information to guide an image processing model to analyze the product image to obtain a target image, wherein the image processing model is obtained by training a text image model, and the image content of the target image is used to simulate the wearing performance result; and displaying the target image on the presentation screen of the VR device or the AR device.

Example 7

An embodiment of the present application may provide an electronic device, which may include a memory and a processor.

Figure 16 is a block diagram of an electronic device according to a method for processing an image in an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and/or required herein.

As shown in FIG. 16 , the device 1600 includes a computing unit 1601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1602 or a computer program loaded from a storage unit 1608 into a random access memory (RAM) 1603. In the RAM 1603, various programs and data required for the operation of the device 1600 can also be stored. The computing unit 1601, the ROM 1602, and the RAM 1603 are connected to each other via a bus 1604. An input/output (I/O) interface 1605 is also connected to the bus 1604.

A number of components in the device 1600 are connected to the I/O interface 1605, including: an input unit 1606, such as a keyboard, a mouse, etc.; an output unit 1604, such as various types of displays, speakers, etc.; a storage unit 1608, such as a disk, an optical disk, etc.; and a communication unit 1609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1609 allows the device 1600 to exchange information/data with other devices through a computer network such as the Internet and/or various telecommunication networks.

The computing unit 1601 may be a variety of general and/or special processing components with processing and computing capabilities. Some examples of the computing unit 1601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1601 performs the various methods and processes described above, such as the image processing method. For example, in some embodiments, the image processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1608. In some embodiments, part or all of the computer program may be loaded and/or installed on the device 1600 via the ROM 1602 and/or the communication unit 1609. When the computer program is loaded into the RAM 1603 and executed by the computing unit 1601, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 1601 may be configured to perform the image processing method in any other appropriate manner (e.g., by means of firmware).

Example 8

The embodiment of the present application further provides a computer program product. Optionally, in the present embodiment, the computer program product may include a computer program, and when the computer program is executed by a processor, the image processing method of the embodiment of the present application is implemented.

According to an embodiment of the present application, a method for processing an image is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

The method embodiment provided in Example 8 of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 17 is a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image processing method according to an embodiment of the present application. As shown in Figure 17, the computer terminal 170 (or mobile device) may include one or more (1702a, 1702b, ..., 1702n are used in the figure to illustrate) processors 1702 (the processor 1702 may include but is not limited to a microprocessor (Microcontroller Unit, referred to as MCU) or a programmable logic device (FieldProgrammable Gate Array, referred to as FPGA) and other processing devices), a memory 1704 for storing data, and a transmission device 1706 for communication functions. In addition, it may also include: a display, an input/output interface (I/O interface), a universal serial bus (Universal Serial Bus, referred to as USB) port (which can be included as one of the ports of the BUS bus), a network interface, a power supply and/or a camera. It can be understood by those skilled in the art that the structure shown in Figure 17 is only for illustration and does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 170 may also include more or fewer components than those shown in FIG. 17 , or have a configuration different from that shown in FIG. 17 .

The hardware structure block diagram shown in Figure 17 can not only serve as an exemplary block diagram of the above-mentioned computer terminal 170 (or mobile device), but also as an exemplary block diagram of the above-mentioned server. In an optional embodiment, Figure 18 shows a block diagram of an embodiment of using the computer terminal 170 (or mobile device) shown in the above-mentioned Figure 17 as a computing node in the computing environment 1801.

FIG18 is a block diagram of a computing environment of an image processing method according to an embodiment of the present application. As shown in FIG18 , a computing environment 1801 includes multiple computing nodes (such as servers) running on a distributed network (shown in the figure using 1810-1, 1810-2, ...). The computing nodes all contain local processing and memory resources, and the terminal user 1802 can remotely run applications or store data in the computing environment 1801. The application can be provided as multiple services 1820-1, 1820-2, 1820-3 and 1820-4 in the computing environment 1601, representing services "F", "G", "I" and "H" respectively.

The end user 1802 can provide and access services through a web browser or other software application on the client, and in some embodiments, the end user 1802's provision and/or request can be provided to the entry gateway 1830. The entry gateway 1830 may include a corresponding agent to handle the provision and/or request for the service (one or more services provided in the computing environment 1801).

Services are provided or deployed based on various virtualization technologies supported by computing environment 1801. In some embodiments, services can be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and/or similar methods. Virtual machine-based virtualization can be to simulate a real computer by initializing a virtual machine, and execute programs and applications without directly contacting any actual hardware resources. While the virtual machine virtualizes the machine, according to container-based virtualization, a container can be started to virtualize the entire operating system so that multiple workloads can run on a single operating system instance.

In an embodiment based on container virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). For example, as shown in FIG. 18 , service 1820-2 can be equipped with one or more Pods 1840-1, 1840-2, ..., 1840-N (collectively referred to as Pods). A Pod may include a proxy 1845 and one or more containers 1842-1, 1842-2, ..., 1842-M (collectively referred to as containers). One or more containers in a Pod process requests related to one or more corresponding functions of the service, and the proxy 1845 generally controls network functions related to the service, such as routing, load balancing, etc. Other services may also be equipped with Pods similar to Pods.

During operation, executing a user request from the end user 1802 may require invoking one or more services in the computing environment 1801, and executing one or more functions of a service may require invoking one or more functions of another service. As shown in FIG18 , service “F” 1820-1 receives a user request from the end user 1802 from the ingress gateway 1830, service “F” 1820-1 may call service “G” 1820-2, and service “G” 1820-2 may request service “I” 1820-3 to execute one or more functions.

The computing environment described above can be a cloud computing environment, where the allocation of resources is managed by the cloud service provider, allowing the development of functions without considering the implementation, adjustment or expansion of servers. The computing environment allows developers to execute code in response to events without building or maintaining complex infrastructure. Services can be divided into a set of functions that can be automatically and independently scaled, rather than expanding a single hardware device to handle potential loads.

Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard parts (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that are executable and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions/operations specified in the flow chart and/or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and/or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions/operations specified in the flow chart and/or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory for short), an optical fiber, a portable compact disc read-only memory (CD-ROM for short), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD), a monitor for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a path ball), through which the user may provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a Local Area Network (LAN), a Wide Area Network (WAN), and the Internet.

A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, read-only memories, random access memories, mobile hard disks, magnetic disks or optical disks.

The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims (21)

1.A method of processing an image, comprising:

Identifying a product image to be processed, wherein the image content of the product image comprises at least one product to be displayed;

Acquiring text description information corresponding to the product to be displayed in the product image, wherein the text description information is used for at least describing the wearing performance result displayed by the product to be displayed when the product to be displayed is worn by a carrier;

And utilizing the text description information to guide an image processing model to analyze the product image to obtain a target image, wherein the image processing model is obtained by training a meridional chart model, and the image content of the target image is used for simulating the wearing performance result.

2. The method of claim 1, wherein using the text description information to direct an image processing model to analyze the product image to obtain a target image comprises:

Extracting text features from the text description information by using the image processing model;

And guiding the image processing model to analyze the product image by utilizing the text characteristics to obtain the target image.

3. The method of claim 2, wherein extracting text features from the text description information using the image processing model comprises:

Extracting the text features from the text description information by using a text feature extraction model in the image processing model;

And guiding the image processing model to analyze the product image by utilizing the text characteristics to obtain the target image, wherein the method comprises the following steps of: and guiding an attention processing model in the image processing model by using the text characteristics, and performing attention processing on the product image to obtain the target image.

4. A method according to claim 3, characterized in that the method further comprises:

Detecting the product image to obtain attribute information of the product to be displayed in the product image;

And guiding an attention processing model in the image processing model by using the text characteristics, and performing attention processing on the product image to obtain the target image, wherein the method comprises the following steps: and guiding the attention processing model by utilizing the text characteristics, and performing attention processing on the attribute information of the product to be displayed to obtain the target image.

5. The method of claim 4, wherein the attention processing model comprises: the first self-attention processing model and the first cross-attention processing model, wherein the text features are utilized to guide the attention processing model to perform attention processing on the attribute information of the product to be displayed, so as to obtain the target image, and the method comprises the following steps:

Performing self-attention processing on the attribute information of the product to be displayed by using the first self-attention processing model to obtain a first self-attention processing result;

Performing cross attention processing on the first self attention processing result and the text feature by using the first cross attention processing model to obtain a first cross attention processing result; the target image is generated based on the first cross-attention processing result.

6. The method of claim 5, wherein the method further comprises:

Carrying out gesture detection on the carrier to obtain gesture information of the carrier, wherein the gesture information is used for representing the gesture of the carrier shown in the simulated wearing performance result;

And performing self-attention processing on the attribute information of the product to be displayed by using the first self-attention processing model to obtain a first self-attention processing result, wherein the first self-attention processing result comprises: and performing self-attention processing on the attribute information and the attitude information of the product to be displayed by using the first self-attention processing model to obtain a first self-attention processing result.

7. The method of claim 6, wherein performing gesture detection on the carrier to obtain gesture information of the carrier comprises:

and calling a gesture coding model to code the gesture of the carrier to obtain skeleton characteristics of the carrier, wherein the gesture information comprises the skeleton characteristics.

8. The method of claim 7, wherein encoding the pose of the carrier using the pose encoding model results in a skeletal feature of the carrier, comprising:

Acquiring an object image, wherein the image content of the object image comprises an object in any posture allowing wearing of the product to be displayed;

And identifying the object in the object image by using the gesture coding model, determining the identified object as the carrier, and performing gesture coding on the carrier to obtain the skeleton characteristic of the carrier.

9. The method of claim 7, wherein the method further comprises:

Residual learning is carried out on the skeleton characteristics by using a first residual network model in the image processing model;

And performing self-attention processing on the attribute information and the gesture information of the product to be displayed by using the first self-attention processing model to obtain a first self-attention processing result, wherein the first self-attention processing result comprises: and performing self-attention processing on the attribute information of the product to be displayed and the learned skeleton characteristics by using the first self-attention processing model to obtain a first self-attention processing result.

10. The method according to claim 9, wherein detecting the product image to obtain the attribute information of the product to be displayed in the product image includes:

calling an image feature extraction model, and extracting original image features from the product image;

Residual learning is carried out on the original image characteristics by using a second residual network model in the image processing model;

performing self-attention processing on the learned original image features by using a second self-attention processing model in the image processing models to obtain a second self-attention processing result, wherein the second self-attention processing result is used for representing image features with the size larger than a size threshold, and the second self-attention processing result is used for representing attribute information of the product to be displayed;

And performing self-attention processing on the attribute information of the product to be displayed and the learned skeleton characteristics by using the first self-attention processing model to obtain a first self-attention processing result, wherein the first self-attention processing result comprises: and receiving the second self-attention processing result output by the second self-attention processing model by using the first self-attention processing model, and performing self-attention processing on the second self-attention processing result and the learned skeleton feature to obtain the first self-attention processing result.

11. The method according to claim 10, wherein the method further comprises:

performing cross attention processing on the second self attention processing result and the text feature by using a second cross attention processing model in the image processing model to obtain a second cross attention processing result;

Determining the second cross-attention processing result as the original image feature, determining a next residual network model of the second residual network model in the image processing models as the second residual network model, determining a next self-attention processing model of the second self-attention processing model in the image processing models as the second self-attention processing model, determining a next cross-attention processing model of the second cross-attention processing model in the image processing models as the second cross-attention processing model, and starting from the following steps until the second residual network model has no next residual network model in the image processing models, the second cross-attention processing model has no next cross-attention processing model in the image processing models:

and carrying out residual learning on the original image features by using a second residual network model in the image processing model.

12. The method of claim 11, wherein the method further comprises:

Determining the first cross-attention processing result as the skeleton feature; determining a next residual network model of the first residual network model in the image processing models as the first residual network model, determining a next self-attention processing model of the first self-attention processing model in the image processing models as the first self-attention processing model, determining a next cross-attention processing model of the first cross-attention processing model in the image processing models as the first cross-attention processing model, and starting from the steps of executing until the first residual network model has no next residual network model in the image processing models, the first self-attention processing model has no next self-attention processing model in the image processing models, the first cross-attention processing model has no next cross-attention processing model in the image processing models:

And residual learning is carried out on the skeleton characteristics by using a first residual network model in the image processing model, wherein the second self-attention processing model corresponds to the first self-attention processing model one by one.

13. The method according to any one of claims 1 to 12, further comprising:

performing image enhancement on the target image;

And outputting the enhanced target image.

14. The method according to any one of claims 1 to 12, wherein the text description information comprises at least one of: and displaying strategy information of the product to be displayed in the displaying process, background content of the target image to be generated and attribute information of the carrier.

15. A method of processing an image, comprising:

Identifying a virtual clothing store deployed on an electronic commerce platform, and identifying an clothing product image to be processed from the virtual clothing store, wherein the image content of the clothing product image comprises at least one clothing product to be displayed;

acquiring text description information corresponding to the clothing product to be displayed in the clothing product image, wherein the text description information is used for at least describing the wearing performance result displayed by the clothing product to be displayed when the clothing product to be displayed is worn by a carrier;

Using the text description information to guide an image processing model to analyze the clothing product image to obtain a target image, wherein the image processing model is obtained by training a meridional chart model, and the image content of the target image is used for simulating the wearing performance result;

And sending the target image to the electronic commerce platform.

16. A method of processing an image, comprising:

responding to image input operation acted on an operation interface, and displaying a product image to be processed on the operation interface, wherein the image content of the product image comprises at least one product to be displayed;

Responding to text input operation acted on the operation interface, and displaying text description information corresponding to the product to be displayed in the product image on the operation interface, wherein the text description information is used for describing at least the wearing performance result displayed by the product to be displayed under the condition of being worn by a carrier;

And responding to image generation operation acted on the operation interface, and displaying a target image matched with the product image and the text description information on the operation interface, wherein the target image is obtained by analyzing the product image by utilizing the text description information and guiding an image processing model, the image processing model is obtained by training a meridional chart model, and the image content of the target image is used for simulating the wearing representation result.

17. A method of processing an image, comprising:

Displaying a product image to be processed on a display picture of Virtual Reality (VR) equipment or Augmented Reality (AR) equipment, wherein the image content of the product image comprises at least one product to be displayed;

Acquiring text description information corresponding to the product to be displayed in the product image, wherein the text description information is used for at least describing the wearing performance result displayed by the product to be displayed when the product to be displayed is worn by a carrier;

Driving the VR equipment or the AR equipment to utilize the text description information to guide an image processing model to analyze the product image so as to obtain a target image, wherein the image processing model is obtained by training a meridional chart model, and the image content of the target image is used for simulating the wearing representation result;

And displaying the target image on a display screen of the VR device or the AR device.

18. An image processing system, comprising:

The system comprises a client, a display device and a display device, wherein the client is used for uploading a product image to be processed, and the image content of the product image comprises at least one product to be displayed; uploading text description information corresponding to the product to be displayed in the product image, wherein the text description information is used for at least describing the wearing performance result displayed by the product to be displayed when the product to be displayed is worn by a carrier;

The server is used for guiding an image processing model to analyze the product image by utilizing the text description information to obtain a target image, wherein the image processing model is obtained by training a meridional chart model, and the image content of the target image is used for simulating the wearing performance result; and sending the target image to the client.

19. An electronic device, comprising:

A memory storing an executable program;

a processor for executing the program, wherein the program when run performs the method of any one of claims 1 to 17.

20. A computer readable storage medium, characterized in that the computer readable storage medium comprises a stored executable program, wherein the executable program when run controls a device in which the computer readable storage medium is located to perform the method of any one of claims 1 to 17.

21. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1 to 17.