CN117008763A - Virtual clothes matching method and image generation model training method - Google Patents

Abstract

This application discloses a virtual clothing matching method and an image generation model training method. Wherein, the method includes: responding to input instructions acting on the operating interface, displaying clothing images of at least two types of virtual clothing on the operating interface; responding to clothing matching instructions acting on the operating interface, displaying at least two types of virtual clothing on the operating interface. A matching image of two types of virtual clothing. The matching image is used to represent an image obtained by matching at least two types of virtual clothing to a virtual object. The matching image is generated using an image generation model corresponding to at least two types of virtual clothing based on at least two types of virtual clothing. Images generated by clothing text corresponding to each type of virtual clothing, and the clothing text is used to describe at least two types of virtual clothing. This application solves the technical problems in related technologies that the effect of virtual clothing matching through virtual fitting technology is poor and the application scenarios are limited.

Description

Virtual clothing matching method, image generation model training method

Technical field

The present application relates to the field of data generation, specifically, to a virtual clothing matching method and an image generation model training method.

Background technique

Traditional virtual fittings are mostly based on 2D clothing pictures. They mainly match the 2D clothing pictures to virtual objects through deformation to obtain the fitting effect. However, the fitting effect generated by the above method is average and cannot be customized. Simulate special styles of clothing.

Contents of the invention

Embodiments of the present application provide a virtual clothing matching method and an image generation model training method to at least solve the technical problems in related technologies that virtual clothing matching through virtual fitting technology has poor effects and limited application scenarios.

According to an aspect of an embodiment of the present application, a virtual clothing matching method is provided, including: responding to an input instruction acting on an operating interface, displaying clothing images of at least two types of virtual clothing on the operating interface; responding to an input instruction acting on an operating interface; The clothing matching instruction on the operation interface displays matching images of at least two types of virtual clothing on the operation interface, where the matching image is used to represent an image obtained by matching at least two types of virtual clothing to a virtual object, and the matching image It is an image generated by using an image generation model corresponding to at least two types of virtual clothing based on clothing text corresponding to at least two types of virtual clothing, and the clothing text is used to describe at least two types of virtual clothing.

According to another aspect of the embodiment of the present application, a virtual clothing matching method is also provided, including: obtaining clothing images of at least two types of virtual clothing; determining image generation models and clothing corresponding to at least two types of virtual clothing. Text, where the clothing text is used to describe at least two types of virtual clothing; an image generation model is used to generate matching images of at least two types of virtual clothing based on the clothing text, where the matching image is used to represent at least two types of virtual clothing. The image obtained by matching the type of virtual clothing to the virtual object.

According to another aspect of the embodiments of the present application, a method for training an image generation model is also provided, including: constructing a training image containing at least two types of virtual clothing; generating training text corresponding to the training image, wherein the training text is Used to describe at least two types of virtual clothing; use training images and training texts to train an initial generation model to obtain an image generation model, wherein the image generation model is used to create clothing text corresponding to at least two types of virtual clothing, Matching images of at least two types of virtual clothing are generated, the clothing text is used to describe the at least two types of virtual clothing, and the matching image is used to represent an image obtained by matching the at least two types of virtual clothing to the virtual object.

According to another aspect of the embodiment of the present application, a method for matching virtual clothing is also provided, including: displaying clothing images of at least two types of virtual clothing on a presentation screen of a virtual reality VR device or an augmented reality AR device; determining Image generation models and clothing texts corresponding to at least two types of virtual clothing, where the clothing text is used to describe at least two types of virtual clothing; the image generation model is used to generate at least two types of virtual clothing based on the clothing text. Matching images, wherein the matching images are used to represent images obtained by matching at least two types of virtual clothing to virtual objects; driving the VR device or AR device to render and display the matching image.

According to another aspect of the embodiment of the present application, a virtual clothing matching method is also provided, including: obtaining clothing images of at least two types of virtual clothing by calling a first interface, where the first interface includes a first parameter, The parameter value of the first parameter is a clothing image; determine the image generation model and clothing text corresponding to at least two types of virtual clothing, where the clothing text is used to describe at least two types of virtual clothing; use the image generation model to generate the clothing based on the clothing Text, generate matching images of at least two types of virtual clothing, where the matching image is used to represent an image obtained by matching at least two types of virtual clothing to a virtual object; output the matching image in the operation interface by calling the second interface , wherein the second interface includes a second parameter, and the parameter value of the second parameter is the matching image.

According to another aspect of the embodiment of the present application, an electronic device is also provided, including: a memory storing an executable program; a processor configured to run the program, wherein when the program is run, any one of the above embodiments is executed. method.

According to another aspect of the embodiment of the present application, a computer-readable storage medium is also provided. The computer-readable storage medium includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute. The method of any one of the above embodiments.

In the embodiment of the present application, in response to a text input instruction acting on the operation interface, clothing images of at least two types of virtual clothing can be displayed on the operation interface, and in response to a clothing matching instruction acting on the operation interface, at least two types of virtual clothing can be determined. The image generation model and clothing text corresponding to each type of virtual clothing are then used to generate images of at least two types of virtual clothing based on the clothing text, and the matching images are displayed on the operation interface, thereby achieving the purpose of virtual fitting. It is easy to notice that since the matching image is an image obtained by matching at least two types of virtual clothing to a virtual object, the purpose of simulating specific styles of clothing and matching and displaying different types of virtual clothing is achieved, and , since the matching image is an image generated based on the clothing text using the image generation model, it achieves the purpose of using the text to generate the image to directly generate a model picture with higher realism and more image elements, thereby improving the virtual clothing matching effect. , expand the technical effects of virtual clothing matching application scenarios, and thus solve the technical problems in related technologies that virtual clothing matching through virtual fitting technology has poor effects and limited application scenarios.

It is easy to notice that the above general description and the following detailed description are only for illustrating and explaining the present application, and do not constitute a limitation of the present application.

Description of the drawings

The drawings described here are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation of the present application. In the attached picture:

Figure 1 is a schematic diagram of the hardware environment of a virtual reality device according to a virtual clothing matching method according to an embodiment of the present application;

Figure 2 is a structural block diagram of a computing environment of a virtual clothing matching method according to an embodiment of the present application;

Figure 3 is a flow chart of a virtual clothing matching method according to Embodiment 1 of the present application;

Figure 4 is a schematic diagram of an optional operation interface according to an embodiment of the present application;

Figure 5 is a flow chart of an optional virtual clothing matching method according to an embodiment of the present application;

Figure 6 is a flow chart of a virtual clothing matching method according to Embodiment 2 of the present application;

Figure 7 is a flow chart of a model training method according to Embodiment 3 of the present application;

Figure 8 is a flow chart of a virtual clothing matching method according to Embodiment 4 of the present application;

Figure 9 is a schematic diagram of a matching result of virtual clothing according to an embodiment of the present application;

Figure 10 is a flow chart of a virtual clothing matching method according to Embodiment 5 of the present application;

Figure 11 is a schematic diagram of a virtual clothing matching device according to Embodiment 6 of the present application;

Figure 12 is a schematic diagram of a virtual clothing matching device according to Embodiment 7 of the present application;

Figure 13 is a schematic diagram of a model training device according to Embodiment 8 of the present application;

Figure 14 is a schematic diagram of a virtual clothing matching device according to Embodiment 9 of the present application;

Figure 15 is a schematic diagram of a virtual clothing matching device according to Embodiment 10 of the present application;

Figure 16 is a structural block diagram of a computer terminal according to an embodiment of the present application.

Detailed ways

In order to enable those in the technical field to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only These are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts should fall within the scope of protection of this application.

It should be noted that the terms "first", "second", etc. in the description and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances so that the embodiments of the application described herein can be practiced in sequences other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus that encompasses a series of steps or units and need not be limited to those explicitly listed. Those steps or elements may instead include other steps or elements not expressly listed or inherent to the process, method, product or apparatus.

First of all, some nouns or terms that appear in the description of the embodiments of this application are applicable to the following explanations:

AIGC: AI Generated Content, artificial intelligence generated content, can also be called generative AI. In this application, it mainly refers to the diffusion model to generate matching images of virtual clothing.

SD: Stable Diffusion, a generative large-scale network model structure.

LORA: Low Rank Adaptation, a small network model used to reduce the cost of fine-tuning large models.

BLIP2: Bootstrapping Language-Image Pre-training, a general, computationally efficient visual-language pre-training model that can leverage frozen pre-trained image encoders and large language models to guide visual language pre-training.

Example 1

According to an embodiment of the present application, a virtual clothing matching method is provided. It should be noted that the steps shown in the flow chart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although A logical order is shown in the flowcharts, but in some cases the steps shown or described may be performed in a different order than herein.

Figure 1 is a schematic diagram of the hardware environment of a virtual reality device according to a virtual clothing matching method according to an embodiment of the present application. As shown in Figure 1, the virtual reality device 104 is connected to the terminal 106, and the terminal 106 and the server 102 are connected through a network. The above-mentioned virtual reality device 104 is not limited to: virtual reality helmets, virtual reality glasses, virtual reality all-in-one machines, etc. The above-mentioned terminal 104 is not limited to a PC, a mobile phone, a tablet, etc. The server 102 may be a server corresponding to a media file operator. The above-mentioned network includes but is not limited to: wide area network, metropolitan area network or local area network.

Optionally, the virtual reality device 104 of this embodiment includes: a memory, a processor, and a transmission device. The memory is used to store an application program, which can be used to perform: in response to input instructions acting on the operating interface, display clothing images of at least two types of virtual clothing on the operating interface; in response to clothing matching acting on the operating interface Instructions to display matching images of at least two types of virtual clothing on the operation interface, wherein the matching image is used to represent an image obtained by matching at least two types of virtual clothing to a virtual object, and the matching image is made by using at least two types of virtual clothing. The image generation model corresponding to the virtual clothing is based on images generated by clothing text corresponding to at least two types of virtual clothing. The clothing text is used to describe at least two types of virtual clothing, thereby solving the problem in related technologies through virtual fitting technology. The technical problems of poor virtual clothing matching and limited application scenarios have been achieved by improving the virtual clothing matching effect and improving the technical effects of virtual clothing matching application scenarios.

The terminal of this embodiment can be used to display clothing images of at least two types of virtual clothing on the presentation screen of a virtual reality (Virtual Reality, referred to as VR) device or an augmented reality (Augmented Reality, referred to as AR) device; determine Image generation models and clothing texts corresponding to at least two types of virtual clothing, where the clothing text is used to describe at least two types of virtual clothing; the image generation model is used to generate at least two types of virtual clothing based on the clothing text. Matching images, where the matching images are used to represent images obtained by matching at least two types of virtual clothing to virtual objects; and sending the matching images to the virtual reality device 104, which after receiving the matching images, places them on the target The location is displayed.

Optionally, the eye-tracking HMD (Head Mount Display) head-mounted display and eye-tracking module provided with the virtual reality device 104 of this embodiment have the same functions as in the above-mentioned embodiment, that is, in the HMD head-mounted display The screen is used to display real-time images, and the eye tracking module in the HMD is used to obtain the real-time movement trajectory of the user's eyeballs. The terminal of this embodiment obtains the user's position information and motion information in the real three-dimensional space through the tracking system, and calculates the three-dimensional coordinates of the user's head in the virtual three-dimensional space, as well as the user's field of view orientation in the virtual three-dimensional space.

The hardware structure block diagram shown in Figure 1 can not only be used as an exemplary block diagram of the above-mentioned AR/VR device (or mobile device), but also can be used as an exemplary block diagram of the above-mentioned server. In an optional embodiment, Figure 2 is a block diagram An embodiment of using the AR/VR device (or mobile device) shown in FIG. 1 above as a computing node in the computing environment 201 is shown. Figure 2 is a structural block diagram of a computing environment of a virtual clothing matching method according to an embodiment of the present application. As shown in Figure 2, the computing environment 201 includes multiple (210-1 is used in the figure) running on a distributed network. 210-2,..., to show) computing nodes (such as servers). Different computing nodes contain local processing and memory resources, and end users 202 can remotely run applications or store data in computing environment 201. Applications may be provided as multiple services 220-1, 220-2, 220-3 and 220-4 in computing environment 201, representing services "A", "D", "E" and "H" respectively.

End users 202 may provide and access services through a web browser or other software application on the client, and in some embodiments, end user 202 offers and/or requests may be provided to ingress gateway 230 . Ingress gateway 230 may include a corresponding agent to handle provisioning and/or requests for services (one or more services provided in computing environment 201).

Services are provided or deployed according to various virtualization technologies supported by computing environment 201. In some embodiments, services may be provided according to virtual machine (VM)-based virtualization, container-based virtualization, and/or similar approaches. Virtual machine-based virtualization can simulate a real computer by initializing a virtual machine to execute programs and applications without directly touching any actual hardware resources. While a virtual machine virtualizes a machine, according to container-based virtualization, containers can be launched to virtualize the entire operating system (OS) so that multiple workloads can run on a single operating system instance.

In one embodiment based on container virtualization, several containers of a service can be assembled into a Pod (for example, Kubernetes Pod). For example, as shown in Figure 2, service 220-2 may be equipped with one or more Pods 240-1, 240-2, ..., 240-N (collectively referred to as Pods). A Pod may include an agent 245 and one or more containers 242-1, 242-2, ..., 242-M (collectively, containers). One or more containers in a Pod handle requests related to one or more corresponding functions of the service. The proxy 245 typically controls network functions related to the service, such as routing, load balancing, etc. Other services can be equipped with similar Pods.

During operation, executing a user request from an end user 202 may require invoking one or more services in the computing environment 201 and executing one or more functions of one service may require invoking one or more functions of another service. As shown in Figure 2, service "A" 220-1 receives the user request of the end user 202 from the ingress gateway 230. Service "A" 220-1 can call service "D" 220-2, and service "D" 220-2 can Service "E" 220-3 is requested to perform one or more functions.

The above-mentioned computing environment can be a cloud computing environment, and the allocation of resources is provided and managed by the cloud service, allowing the development of functions without considering the implementation, adjustment or expansion of the server. This computing environment allows developers to execute code that responds to events without building or maintaining complex infrastructure. Instead of scaling a single hardware device to handle the potential load, services can be partitioned into a set of functions that can automatically scale independently.

Under the above operating environment, this application provides a virtual clothing matching method as shown in Figure 3. It should be noted that the virtual clothing matching method of this embodiment can be executed by the mobile terminal of the embodiment shown in FIG. 1 . Figure 3 is a flow chart of a virtual clothing matching method according to Embodiment 1 of the present application. As shown in Figure 3, the method may include the following steps:

Step S302: In response to an input instruction acting on the operation interface, clothing images of at least two types of virtual clothing are displayed on the operation interface.

The above-mentioned operation interface may be an interface provided by the mobile terminal for users to perform human-computer interaction. For example, it may be the operation interface as shown in Figure 4, but it is not limited to this. It may also be other operation interfaces, which can be performed by the user according to actual needs. set up. The above-mentioned input instructions can be instructions generated by the user selecting clothing images on the operation interface. For example, taking the operation interface shown in Figure 4 as an example, the input instructions can be the user selecting the items they wish to display in the "clothing display area". Instructions generated by multiple virtual garments.

The above-mentioned at least two types of virtual clothing can be virtual clothing that the user wants to display together. The types here can include: tops, bottoms, shoes, hats, gloves, accessories, bags, umbrellas, etc., but are not limited to these. In the embodiment of the present application, two types of virtual clothing, tops and bottoms, are used as examples for description, but it is not limited to this.

In an optional embodiment, the user can operate on the operation interface provided by the mobile terminal to select multiple virtual garments that he or she wishes to display together, thereby generating an input instruction. After receiving the input command, the mobile terminal device can mark the clothing images of multiple virtual clothing selected by the user on the operation interface, and the user can confirm whether there are errors in the clothing images.

For example, taking the generation requirements on the seller side in the e-commerce scenario as an example, in order to be able to generate model pictures for their products that are comparable to professional photography, the seller can enter the "clothing display area" in the operation interface as shown in Figure 4. Display the clothing products sold by the seller, so that the seller can select multiple clothing products in the "Clothing Display Area". For example, if the seller wants to display a green shirt and blue jeans together, he can select the green shirt in the "Clothing Display Area" image and the image of blue jeans, so that the mobile terminal can generate input instructions and mark the image selected by the seller, for example, add a "√" mark to the image selected by the seller in the "clothing display area" for the seller to view .

For another example, take the generation of demand on the buyer's side in an e-commerce scenario as an example. In the process of purchasing clothing, in order to view the matching display effect of multiple clothing to be purchased that the buyer wants to display together, the buyer can In the "Clothing Display Area" in the operation interface as shown in Figure 4, select multiple garments to be purchased that you want to display in combination. For example, if a buyer wants to see the matching display effect of a green shirt and blue jeans, he can go to "Clothing" Select the image of a green shirt and the image of blue jeans in the "display area", so that the mobile terminal can generate an input instruction and mark the image selected by the buyer, for example, mark the image selected by the buyer in the "clothing display area" Add a "√" mark for buyers to view.

Step S304: In response to the clothing matching instruction acting on the operation interface, the matching images of at least two types of virtual clothing are displayed on the operation interface, where the matching images are used to represent the matching of at least two types of virtual clothing to the virtual object. The obtained image, the matching image is an image generated by using an image generation model corresponding to at least two types of virtual clothing based on clothing text corresponding to at least two types of virtual clothing, and the clothing text is used to describe at least two types of virtual clothing. .

The above clothing matching instruction may be an instruction to match and display at least two types of virtual clothing. Alternatively, it may be an instruction generated by the user clicking a preset control on the operation interface, for example, as shown in Figure 4 Taking the operation interface as an example, the clothing matching instructions can be instructions generated by the user clicking the "Virtual Fitting" button.

The above-mentioned matching image can be a highly realistic model image generated by matching at least two types of virtual clothing and wearing it on a virtual object. The above-mentioned virtual object can be a preset virtual model, or it can be a virtual model with the user. Matching avatars, but not only. The above image generation model can be a large model, for example, it can be an SD model, but it is not limited to this. Due to the large number of parameters of the image generation model, in order to avoid the difficulty and high cost of training a large model, a low-rank model can be pre-trained and the two models can be run in parallel to achieve the large model and the low-rank model. In the common inference process, the low-rank model here can be a model with a smaller network structure, such as the LORA model, but is not limited to this.

In an optional embodiment, when the user confirms that the clothing image is correct, the user can operate on the operation interface and click the preset control to generate a clothing matching instruction, so that the mobile terminal can send the clothing image confirmed by the user. To the server, the server determines an image generation model that can match and display at least two types of virtual clothing, and uses clothing text to describe at least two types of virtual clothing, and uses the image generation model to generate a matching corresponding to the clothing text. The image is returned to the mobile terminal, so that the mobile terminal can display the matching image on the operation interface for the user to view.

For example, taking the generated demand on the seller side in an e-commerce scenario as an example, when the seller determines that multiple clothing products are correct, the seller can click the "virtual fitting" button in the operation interface as shown in Figure 4, thereby The mobile terminal can receive the clothing matching instruction and send images of multiple clothing products to the server, and the server determines an image generation model that can display the multiple clothing products together, as well as the description information of the multiple clothing products (i.e., the above-mentioned clothing text), use the image generation model to generate a model image corresponding to the description information, that is, the above-mentioned matching image, and return it to the mobile terminal, which is displayed in the "clothing matching display area" in the operation interface, so that the seller can view to generate model pictures for your own clothing products, thereby greatly improving the quality of product promotion while ensuring low costs.

For another example, take the generation of demand on the buyer's side in an e-commerce scenario as an example. When the buyer determines that the multiple garments to be purchased are correct, the buyer can click "Virtual Trial" in the operation interface as shown in Figure 4. "Clothes" button, so that the mobile terminal can receive clothing matching instructions and send multiple images of clothing to be purchased to the server, and the server determines an image generation model that can match and display multiple clothing to be purchased, as well as multiple images to be purchased. The description information of the clothing (i.e., the above-mentioned clothing text) uses the image generation model to generate the model image corresponding to the description information, that is, the above-mentioned matching image, and returns it to the mobile terminal, and the mobile terminal displays the "clothing matching display" in the operation interface. "area", so that buyers can see the matching display effect after wearing multiple clothing to be purchased, thereby providing buyers with a reference for purchasing clothing.

Through the solutions provided by the above embodiments of the present application, clothing images of at least two types of virtual clothing can be displayed on the operating interface in response to text input instructions acting on the operating interface. In response to clothing matching instructions acting on the operating interface, clothing images of at least two types of virtual clothing can be displayed on the operating interface. Determine the image generation model and clothing text corresponding to at least two types of virtual clothing, and then use the image generation model to generate at least two types of virtual clothing images based on the clothing text, and display the matching images on the operation interface, thereby realizing virtual fitting the goal of. It is easy to notice that since the matching image is an image obtained by matching at least two types of virtual clothing to a virtual object, the purpose of simulating specific styles of clothing and matching and displaying different types of virtual clothing is achieved, and , since the matching image is an image generated based on the clothing text using the image generation model, it achieves the purpose of using the text to generate the image to directly generate a model picture with higher realism and more image elements, thereby improving the virtual clothing matching effect. , expand the technical effects of virtual clothing matching application scenarios, and thus solve the technical problems in related technologies that virtual clothing matching through virtual fitting technology has poor effects and limited application scenarios.

In the above embodiment of the present application, the method further includes: using the feature extraction module in the image generation model to extract features from the clothing text to obtain clothing features; using the generation module in the image generation model to generate matching based on clothing features and object features. Images where object features are used to characterize virtual objects.

The above-mentioned feature extraction module may be an Encoder (encoder) in the image generation model, and the above-mentioned generation module may include UNet (U-shaped convolutional neural network) and Decoder (decoder) in the image generation model, but it is not limited to this.

The above-mentioned object features may be features matching virtual objects contained in the image. If the user does not input description information to describe the virtual object, the characteristics of a preset virtual object can be used as the above-mentioned object characteristics; if the user inputs the description information to describe the virtual object, then the characteristics of the virtual object can be The description information is used for feature extraction, and the features of the virtual object are extracted as the above-mentioned object features.

In an optional embodiment, the feature extraction module can be used to extract clothing features, which may include but are not limited to: clothing color, clothing material, clothing style, etc. Furthermore, the generation module can be used to process clothing features and object features to achieve the purpose of the Vincentian graph, thereby generating matching images.

In the above embodiments of the present application, the object text used to describe the virtual object is also displayed on the operation interface. The method further includes: using a feature extraction module to extract features from the object text to obtain object features.

In an optional embodiment, in order to ensure that the matching image can meet the user's virtual fitting requirements, the user can specify the virtual object in the matching image, that is, while selecting multiple virtual clothes, the user can enter the object text , for example, the user can input "a standing girl with short hair" as the object text, so that the object features can be obtained by separately extracting features from the object text, and then the generation module can be used to generate a virtual object containing multiple virtual clothes. matching images.

In the above embodiments of the present application, the operating interface also displays environmental text describing the virtual environment in which the virtual object is located. The generation module in the image generation model is used to generate matching images based on clothing features and object features, including: using The feature extraction module extracts features from the environmental text to obtain environmental features; the generation module is used to generate matching images based on clothing features, object features and environmental features.

The above-mentioned virtual environment can be a background that is desired to be displayed in the matching image, such as walls, floors, windows, etc., but is not limited to this.

In an optional embodiment, in order to highlight the display effect of the virtual fitting, the user can specify the background elements in the matching image, that is, while selecting multiple virtual clothes, the user can input environmental text, for example, The user can input "simple background" as the environment text, and then the clothing features can be obtained through feature extraction of the environment text, and then the generation module can be used to generate matching images containing virtual objects and backgrounds wearing multiple virtual clothing.

In the above embodiments of the present application, the image generation model includes: a trained generation model and a large model. The large model includes at least one first module connected in sequence. The trained generation model includes at least one second module. The second module is connected with The method further includes: using the current first module in at least one first module to generate first matching data based on the clothing text; using the current second module in at least one second module to generate based on the clothing text. second matching data, where the current second module is the second module corresponding to the current first module; merge the first matching data and the second matching data to obtain merged data, where the current first module is at least one In the case of the last module in the first module, the merged data is a matching image; in the case that the current first module is not the last module in at least one first module, the merged data is input to at least one first module respectively. the next first module in and the next second module in at least one second module.

The above-mentioned large model is composed of multiple network modules (i.e., the above-mentioned first module) connected in sequence. In order to avoid the high cost of parameter adjustment of the large model, a low-rank model (i.e., the above-mentioned well-trained model) can be pre-trained. Generative model), the network modules contained in the low-rank model correspond to the network modules contained in the large model. The above-mentioned first matching data and second matching data may be feature vectors generated during model inference.

In an optional embodiment, based on the clothing text, a single network module of the large model generates the first matching data, the corresponding network module in the low-rank model generates the second matching data, and then the first matching data and the second matching data are Merge is performed, and the merged data continues to be passed to the next sub-network module, and finally matching images of at least two types of virtual clothing are obtained.

In the above embodiment of the present application, the method further includes: constructing a training image containing at least two types of virtual clothing; and generating training text corresponding to the training image, wherein the training text is used to describe at least two types of virtual clothing. ; Use training images and training text to train the initial generation model to obtain a trained generation model.

In an optional embodiment, since the matching image not only needs to show the effect of a piece of clothing worn on the model, but also often needs to show the effect of different types of clothing worn on the model, therefore, the matching image that needs to be displayed can be pre-constructed. Images of different types of virtual clothing worn on models respectively, which are the above-mentioned training images. Furthermore, the corresponding training text can be manually generated based on the training image as the label of the training image, or an image text reasoning model can be used to realize the purpose of drawing text and the corresponding training text can be generated as the label of the training image. Finally, the training images and the labels of the training images are used as the final training data to train the initial generation model.

It should be noted that in order to reduce the cost of generating matching images, a smaller number of training images can be provided for training. The number of training images here can be 6, but is not limited to this.

In the above embodiments of the present application, constructing a training image containing at least two types of virtual clothing includes: obtaining an original image, wherein the display content of the original image contains at least two types of virtual clothing and at least one object; Segment the area where at least one object is located to obtain a segmented image; superimpose the segmented image onto a preset image to generate a training image.

The above-mentioned original image may be an image obtained through professional photography of models wearing different types of virtual clothing, and the original image may contain one or more models (ie, at least one of the above-mentioned objects).

The above-mentioned preset image can be an image with a preset background. In order to avoid the influence of background elements on matching image generation, the background in the preset image can be a solid color background and does not contain any background elements, but it is not limited to this.

In an optional embodiment, in order to facilitate subsequent generation of the background in the matching image based on the background elements input by the user, the image used as the training image requires a clean background and a clear subject. Therefore, the clothing image specified by the user can be obtained as the original image, or the preset clothing image can be obtained as the original image. The original image usually contains multiple virtual clothing worn on the model. Therefore, the area where the model is located can be Segmentation, and can segment different types of virtual clothing to prevent the inherent clothing matching in the original image from affecting the matching image. Further, after cropping the segmented image to a specified size, the segmented image can be pasted on a solid color background to construct a training image.

In the above embodiment of the present application, obtaining the original image includes: displaying at least one image set in the operation interface, wherein different image sets contain at least two types of virtual clothing that are different; responding to a selection instruction acting on at least one image set, Get the image collection corresponding to the selection instruction and get the original image.

The above-mentioned at least one image collection may be product model pictures of different sellers in an e-commerce scenario, or may be product model pictures of different brands obtained from the Internet, but is not limited to this.

The above-mentioned selection instructions may be instructions generated by the user clicking on at least one image set, but are not limited to this.

In an optional embodiment, in order to ensure that the generated matching images better meet the user's requirements, the user can specify a product model image of a specific brand, or specify a product model image of a specific seller to construct the final training image. Optionally, at least one image set can be displayed in the operation interface of the mobile terminal for viewing. The user selects one or more image sets from the at least one image set according to matching image generation requirements, and the user directly clicks on the selected image set. to generate a selection instruction, so that the mobile terminal can determine the image set selected by the user based on the selection instruction and obtain the original image.

In the above embodiments of the present application, generating training text corresponding to the training image includes: using an image text reasoning model to perform text prediction on the training image and generating training text.

The above image text reasoning model may be a model capable of generating text based on images, for example, it may be BLIP2, but is not limited to this.

In an optional embodiment, in order to reduce the user's cost of labeling training images, a pre-trained image text reasoning model can be used to perform text prediction on the training images to obtain the training text.

The following is a detailed explanation based on Figure 5, taking the matching display of tops and bottoms in an e-commerce scenario as an example. As shown in Figure 5, the solution is mainly divided into three processes, namely clothing data preparation, LORA model training and model inference. Among them, in the clothing data preparation process, the training data for training the LORA model needs to be prepared in advance. The training data mainly includes clothing images and text labels, that is, it includes training images and training text. You can input the original image of the specified clothing, and then perform human body segmentation through the human body detection model, segment the human body parts, and paste them on a solid color background to construct a training image, and then use the image text inference model to automatically label the training image and generate the corresponding label description. , for example, the tag description generated for the tops can be "the upper body of a girl, wearing a top, simple background", the tag description generated for the bottoms can be "the lower body of a girl, wearing jeans, a simple background" and so on. In the LORA model training process, you can obtain a pre-trained SD large model. This model has the ability to generate high-quality clothing model images, that is, to generate matching images. You can use the training data to train a LORA model, and combine the LORA model with the The models are processed in parallel to achieve specified matching image generation. In the model inference process, you can guide based on the text data corresponding to upper and lower clothing (including clothing text, object text and environment text), for example, "a standing girl with short hair, wearing a green shirt and blue jeans, with a simple background" The SD model cooperates with the trained LORA model to generate a matching image, that is, a model image wearing specified tops and bottoms is generated.

It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all It is information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data need to comply with the relevant laws, regulations and standards of relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or reject.

It should be noted that for the sake of simple description, the foregoing method embodiments are expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited by the described action sequence. Because in accordance with this application, certain steps may be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily necessary for this application.

Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or that contributes to the existing technology. The computer software product is stored in a storage medium (such as ROM/RAM, disk, CD), including several instructions to cause a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in various embodiments of this application.

Example 2

According to an embodiment of the present application, a virtual clothing matching method is also provided. It should be noted that the steps shown in the flow chart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although A logical order is shown in the flowcharts, but in some cases, the steps shown or described may be performed in a different order than herein.

Figure 6 is a flow chart of a virtual clothing matching method according to Embodiment 2 of the present application. As shown in Figure 6, the method may include the following steps:

Step S602: Obtain clothing images of at least two types of virtual clothing.

Step S604: Determine image generation models and clothing text corresponding to at least two types of virtual clothing, where the clothing text is used to describe at least two types of virtual clothing.

Step S606: Use an image generation model to generate matching images of at least two types of virtual clothing based on clothing text, where the matching images are used to represent images obtained by matching at least two types of virtual clothing to virtual objects.

In an optional embodiment, the user can select clothing images of at least two types of virtual clothing on the mobile terminal, and the mobile terminal sends the clothing images to the server. The server can obtain clothing images sent by the mobile terminal, determine an image generation model that can match and display at least two types of virtual clothing, and use clothing text to describe at least two types of virtual clothing, using the image generation model based on The clothing text generates matching images of at least two types of virtual clothing, so that the server can return the matching images to the mobile terminal, and the mobile terminal displays them for the user to view.

It should be noted that the preferred implementations involved in the above embodiments of the present application are the same as the solutions, application scenarios, and implementation processes provided in Embodiment 1, but are not limited to the solutions provided in Embodiment 1.

Example 3

According to the embodiment of the present application, a model training method is also provided. It should be noted that the steps shown in the flow chart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although in the flow A logical order is shown in the figures, but in some cases the steps shown or described may be performed in a different order than herein.

Figure 7 is a flow chart of a model training method according to Embodiment 3 of the present application. As shown in Figure 7, the method may include the following steps:

Step S702: Construct training images containing at least two types of virtual clothing.

Step S704: Generate training text corresponding to the training image, where the training text is used to describe at least two types of virtual clothing.

Step S706, use the training image and the training text to train the initial generation model to obtain an image generation model, wherein the image generation model is used to generate at least two types of virtual clothing based on clothing text corresponding to at least two types of virtual clothing. The matching image and clothing text are used to describe at least two types of virtual clothing, and the matching image is used to represent an image obtained by matching at least two types of virtual clothing to a virtual object.

In an optional embodiment, an image generation model can be pre-trained by the server. Specifically, images of different types of virtual clothing that need to be displayed and worn on the model, that is, the above-mentioned training images, can be pre-constructed. Furthermore, the corresponding training text can be manually generated based on the training image as the label of the training image, or an image text reasoning model can be used to realize the purpose of drawing text and the corresponding training text can be generated as the label of the training image. Finally, the training images and the labels of the training images are used as the final training data to train the image generation model.

It should be noted that the preferred implementations involved in the above embodiments of the present application are the same as the solutions, application scenarios, and implementation processes provided in Embodiment 1, but are not limited to the solutions provided in Embodiment 1.

Example 4

According to the embodiments of the present application, a matching method for virtual clothing that can be applied to virtual reality VR equipment, augmented reality AR equipment and other virtual reality scenarios is also provided. It should be noted that the steps shown in the flow chart of the accompanying drawings are may be executed in a computer system, such as a set of computer-executable instructions, and, although a logical sequence is shown in the flowchart diagrams, in some cases, the steps shown or described may be executed in a sequence different from that described herein. step.

Figure 8 is a flow chart of a virtual clothing matching method according to Embodiment 4 of the present application. As shown in Figure 8, the method may include the following steps:

Step S802: Display clothing images of at least two types of virtual clothing on the presentation screen of the virtual reality VR device or the augmented reality AR device.

Step S804: Determine image generation models and clothing text corresponding to at least two types of virtual clothing, where the clothing text is used to describe at least two types of virtual clothing.

Step S806: Use an image generation model to generate matching images of at least two types of virtual clothing based on clothing text, where the matching images are used to represent images obtained by matching at least two types of virtual clothing to virtual objects.

Step S808: Drive the VR device or AR device to render and display the matching image.

Optionally, in this embodiment, the above-mentioned matching method of virtual clothing can be applied to a hardware environment composed of a server and a virtual reality device. The matching image of the virtual clothing is displayed on the presentation screen of the virtual reality VR device or the augmented reality AR device. The server can be a server corresponding to the media file operator. The above-mentioned network includes but is not limited to: wide area network, metropolitan area network or local area network. The above-mentioned virtual reality Equipment is not limited to: virtual reality helmets, virtual reality glasses, virtual reality all-in-one machines, etc.

Optionally, the virtual reality device includes: a memory, a processor and a transmission device. The memory is used to store an application program, which can be used to perform: display clothing images of at least two types of virtual clothing on the presentation screen of the virtual reality VR device or the augmented reality AR device; determine the correspondence between at least two types of virtual clothing The image generation model and clothing text are used, where the clothing text is used to describe at least two types of virtual clothing; the image generation model is used to generate matching images of at least two types of virtual clothing based on the clothing text, where the matching image is used To represent the image obtained by matching at least two types of virtual clothing to the virtual object; driving the VR device or AR device to render and display the matching image.

It should be noted that the above-mentioned matching method of virtual clothing applied in a VR device or AR device in this embodiment may include the method of the embodiment shown in FIG. 3 to achieve the purpose of the matching method of driving the VR device or AR device to display virtual clothing. .

Optionally, the processor of this embodiment can call the application program stored in the above memory through the transmission device to perform the above steps. The transmission device can receive media files sent by the server through the network, and can also be used for data transmission between the above-mentioned processor and the memory.

Optionally, in the virtual reality device, a head-mounted display with eye tracking is used. The screen in the HMD is used to display the displayed video picture, and the eye-tracking module in the HMD is used to obtain real-time images of the user's eyeballs. Motion trajectory, tracking system, used to track the user's position information and motion information in the real three-dimensional space, calculation processing unit, used to obtain the user's real-time position and motion information from the tracking system, and calculate the position of the user's head in the virtual three-dimensional space The three-dimensional coordinates in the virtual three-dimensional space, as well as the user's visual field direction in the virtual three-dimensional space, etc.

In the embodiment of the present application, the virtual reality device can be connected to the terminal, and the terminal and the server are connected through the network. The above-mentioned virtual reality device is not limited to: virtual reality helmets, virtual reality glasses, virtual reality all-in-one machines, etc., and the above-mentioned terminals are not limited to: It is not limited to PCs, mobile phones, tablets, etc., the server can be a server corresponding to the media file operator, and the above-mentioned network includes but is not limited to: wide area network, metropolitan area network or local area network.

Figure 9 is a schematic diagram of a matching result of virtual clothing according to an embodiment of the present application. As shown in Figure 9, clothing images of at least two types of virtual clothing are displayed on the presentation screen of the virtual reality VR device or the augmented reality AR device. , the image generation model corresponding to at least two types of virtual clothing can be used to generate clothing text corresponding to at least two types of virtual clothing, such as "1 standing girl with short hair, wearing a green shirt and blue jeans, simple background", Generate matching images of at least two types of virtual clothing, and drive the virtual reality VR device or augmented reality AR device to render and display the matching images, such as the model image shown in Figure 9.

It should be noted that the preferred implementations involved in the above embodiments of the present application are the same as the solutions, application scenarios, and implementation processes provided in Embodiment 1, but are not limited to the solutions provided in Embodiment 1.

Example 5

According to an embodiment of the present application, a virtual clothing matching method is also provided. It should be noted that the steps shown in the flow chart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although A logical order is shown in the flowcharts, but in some cases, the steps shown or described may be performed in a different order than herein.

Figure 10 is a flow chart of a virtual clothing matching method according to Embodiment 5 of the present application. As shown in Figure 10, the method may include the following steps:

Step S1002: Obtain clothing images of at least two types of virtual clothing by calling a first interface, where the first interface includes a first parameter, and the parameter value of the first parameter is the clothing image.

The above-mentioned first interface may be a physical interface or a virtual interface that connects the mobile terminal to the server. This interface may realize the purpose of transmitting clothing images from the mobile terminal to the server.

Step S1004: Determine image generation models and clothing text corresponding to at least two types of virtual clothing, where the clothing text is used to describe at least two types of virtual clothing.

Step S1006: Use an image generation model to generate matching images of at least two types of virtual clothing based on clothing text, where the matching images are used to represent images obtained by matching at least two types of virtual clothing to virtual objects.

Step S1008: Output the matching image in the operation interface by calling a second interface, where the second interface includes a second parameter, and the parameter value of the second parameter is the matching image.

The above-mentioned first interface may be a physical interface or a virtual interface that connects the mobile terminal to the server. The interface may realize the purpose of transmitting the matching image from the server to the mobile terminal.

It should be noted that the preferred implementations involved in the above embodiments of the present application are the same as the solutions, application scenarios, and implementation processes provided in Embodiment 1, but are not limited to the solutions provided in Embodiment 1.

Example 6

According to an embodiment of the present application, a virtual clothing matching device for implementing the above virtual clothing matching method is also provided. As shown in Figure 11, the device 1100 includes: a first display module 1102 and a second display module 1104.

The first display module 1102 is used to display clothing images of at least two types of virtual clothing on the operation interface in response to text input instructions acting on the operation interface; the second display module 1104 is used to respond to the text input instructions acting on the operation interface. Clothing matching instructions display matching images of at least two types of virtual clothing on the operation interface, where the matching image is used to represent an image obtained by matching at least two types of virtual clothing to a virtual object, and the matching image is made by using at least two types of virtual clothing. The image generation model corresponding to each type of virtual clothing generates images based on clothing text corresponding to at least two types of virtual clothing, and the clothing text is used to describe at least two types of virtual clothing.

It should be noted here that the first display module 1102 and the second display module 1104 correspond to steps S302 to S304 in Embodiment 1. The examples and application scenarios implemented by the two modules and the corresponding steps are the same, but they are not the same. It is limited to the content disclosed in the above-mentioned Embodiment 1. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory and processed by one or more processors. The above-mentioned modules may also be part of a device that can run on the AR/VR provided in Embodiment 1. in the device.

In the above embodiment of the present application, the device further includes: a feature extraction module and an image generation module.

Among them, the feature extraction module is used to use the feature extraction module in the image generation model to extract features from the clothing text to obtain clothing features; the image generation module is used to use the generation module in the image generation model to generate matching images based on clothing features and object features. , where object features are used to characterize virtual objects.

In the above embodiments of the present application, the object text used to describe the virtual object is also displayed on the operation interface, and the feature extraction module is also used to extract features from the object text using the feature extraction module to obtain object features.

In the above embodiments of the present application, the operating interface also displays environmental text for describing the virtual environment in which the virtual object is located, and the feature extraction module is also used to extract features from the environmental text using the feature extraction module to obtain environmental features; image The generation module is also used to generate matching images based on clothing features, object features and environment features using the generation module.

In the above embodiments of the present application, the image generation model includes: a trained generation model and a large model. The large model includes at least one first module connected in sequence. The trained generation model includes at least one second module. The second module is connected with The first module matches, and the device further includes: a first generation module, a second generation module, a merging module and an input module.

Wherein, the first generation module is used to generate the first matching data based on the clothing text by using the current first module in the at least one first module, wherein the current first module is the first module in the at least one first module. In this case; the second generation module is used to generate second matching data based on the clothing text using the current second module in at least one second module, where the current second module is the second module corresponding to the current first module; merge The module is used to merge the first matching data and the second matching data to obtain merged data, wherein, when the current first module is the last module of at least one first module, the merged data is a matching image; the input module Used to respectively input merged data to the next first module in the at least one first module and the next module in the at least one second module when the current first module is not the last module in the at least one first module. A second module.

In the above embodiment of the present application, the device further includes: a building module, a text generation module and a training module.

Wherein, the building module is used to construct a training image containing at least two types of virtual clothing; the text generation module is used to generate training text corresponding to the training image, wherein the training text is used to describe at least two types of virtual clothing; training The module is used to train the initial generation model using training images and training text to obtain a trained generation model.

In the above embodiments of the present application, the building modules include: an acquisition unit, a segmentation unit and an overlay unit.

Wherein, the acquisition unit is used to acquire the original image, wherein the display content of the original image includes at least two types of virtual clothing and at least one object; the segmentation unit is used to segment the area where at least one object is located in the original image to obtain the segmented image; The overlay unit is used to superimpose the segmented images onto the preset images to generate training images.

In the above embodiments of the present application, the acquisition unit is also configured to display at least one image set in the operation interface, wherein different image sets contain at least two types of virtual clothing that are different; in response to a selection instruction acting on at least one image set, Get the image collection corresponding to the selection instruction and get the original image.

In the above embodiments of this application, the text generation module includes: a generation unit.

Among them, the generation unit is used to use the image text reasoning model to perform text prediction on training images and generate training text.

It should be noted that the preferred implementations involved in the above embodiments of the present application are the same as the solutions, application scenarios, and implementation processes provided in Embodiment 1, but are not limited to the solutions provided in Embodiment 1.

Example 7

According to an embodiment of the present application, a virtual clothing matching device for implementing the above virtual clothing matching method is also provided. As shown in Figure 12, the device 1200 includes: an acquisition module 1202, a determination module 1204 and a generation module 1206.

Among them, the obtaining module 1202 is used to obtain clothing images of at least two types of virtual clothing; the determining module 1204 is used to determine image generation models and clothing text corresponding to at least two types of virtual clothing, wherein the clothing text is used to generate at least two types of virtual clothing. Each type of virtual clothing is described; the generation module 1206 is configured to use an image generation model to generate matching images of at least two types of virtual clothing based on the clothing text, where the matching images are used to represent matching at least two types of virtual clothing to The resulting image of the virtual object.

It should be noted here that the above-mentioned acquisition module 1202, determination module 1204 and generation module 1206 correspond to steps S602 to step S606 in Embodiment 2. The examples and application scenarios implemented by the three modules and the corresponding steps are the same, but they are not the same. It is limited to the content disclosed in the above-mentioned Embodiment 2. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory and processed by one or more processors. The above-mentioned modules may also be part of a device that can run on the AR/VR provided in Embodiment 1. in the device.

It should be noted that the preferred implementations involved in the above embodiments of the present application are the same as the solutions, application scenarios, and implementation processes provided in Embodiment 1, but are not limited to the solutions provided in Embodiment 1.

Example 8

According to an embodiment of the present application, a model training device for implementing the above model training method is also provided. As shown in Figure 13, the device 1300 includes: a construction module 1302, a generation module 1304 and a training module 1306.

Among them, the construction module 1302 is used to construct a training image containing at least two types of virtual clothing; the generation module 1304 is used to generate training text corresponding to the training image, wherein the training text is used to describe at least two types of virtual clothing; The training module 1306 is used to train the initial generation model using training images and training text to obtain an image generation model, wherein the image generation model is used to generate at least two types of virtual clothing based on clothing text corresponding to at least two types of virtual clothing. Matching images of clothing, clothing text is used to describe at least two types of virtual clothing, and matching images are used to represent images obtained by matching at least two types of virtual clothing to virtual objects.

It should be noted here that the above-mentioned building module 1302, generation module 1304 and training module 1306 correspond to step S702 to step S706 in Embodiment 3. The examples and application scenarios implemented by the three modules and the corresponding steps are the same, but they are not the same. It is limited to the content disclosed in the above-mentioned Embodiment 3. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory and processed by one or more processors. The above-mentioned modules may also be part of a device that can run on the AR/VR provided in Embodiment 1. in the device.

It should be noted that the preferred implementations involved in the above embodiments of the present application are the same as the solutions, application scenarios, and implementation processes provided in Embodiment 1, but are not limited to the solutions provided in Embodiment 1.

Example 9

According to an embodiment of the present application, a virtual clothing matching device for implementing the above virtual clothing matching method is also provided. As shown in Figure 14, the device 1400 includes: a first display module 1402, a determination module 1404, and a generation module. 1406 and the second display module 1408.

Among them, the first display module 1402 is used to display clothing images of at least two types of virtual clothing on the presentation screen of the virtual reality VR device or the augmented reality AR device; the determination module 1404 is used to determine the clothing images corresponding to the at least two types of virtual clothing. The image generation model and clothing text, where the clothing text is used to describe at least two types of virtual clothing; the generation module 1406 is used to use the image generation model to generate matching images of at least two types of virtual clothing based on the clothing text, where , the matching image is used to represent the image obtained by matching at least two types of virtual clothing to the virtual object; the second display module 1408 is used to drive the VR device or AR device to render and display the matching image.

It should be noted here that the above-mentioned first display module 1402, determination module 1404, generation module 1406 and second display module 1408 correspond to steps S802 to step S808 in Embodiment 4. The four modules and the corresponding steps implement The examples and application scenarios are the same, but are not limited to the content disclosed in the above-mentioned Embodiment 4. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory and processed by one or more processors. The above-mentioned modules may also be part of a device that can run on the AR/VR provided in Embodiment 1. in the device.

It should be noted that the preferred implementations involved in the above embodiments of the present application are the same as the solutions, application scenarios, and implementation processes provided in Embodiment 1, but are not limited to the solutions provided in Embodiment 1.

Example 10

According to an embodiment of the present application, a virtual clothing matching device for implementing the above virtual clothing matching method is also provided. As shown in Figure 15, the device 1500 includes: a first calling module 1502, a determination module 1504, and a generation module. 1506 and the second calling module 1508.

The first calling module 1502 is used to obtain clothing images of at least two types of virtual clothing by calling a first interface, where the first interface includes a first parameter, and the parameter value of the first parameter is the clothing image; the determination module 1504 uses The image generation model and clothing text corresponding to at least two types of virtual clothing are determined, wherein the clothing text is used to describe the at least two types of virtual clothing; the generation module 1506 is used to use the image generation model to generate at least one based on the clothing text. Matching images of two types of virtual clothing, where the matching image is used to represent an image obtained by matching at least two types of virtual clothing to a virtual object; the second calling module 1508 is used to call the second interface in the operation interface The matching image is output, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the matching image.

It should be noted here that the above-mentioned first calling module 1502, determining module 1504, generating module 1506 and second calling module 1508 correspond to steps S1002 to step S1008 in Embodiment 5. The four modules and the corresponding steps implement The examples and application scenarios are the same, but are not limited to the content disclosed in the above-mentioned Embodiment 5. It should be noted that the above-mentioned modules or units may be hardware components or software components stored in a memory and processed by one or more processors. The above-mentioned modules may also be part of a device that can run on the AR/VR provided in Embodiment 1. in the device.

It should be noted that the preferred implementations involved in the above embodiments of the present application are the same as the solutions, application scenarios, and implementation processes provided in Embodiment 1, but are not limited to the solutions provided in Embodiment 1.

Example 11

Embodiments of the present application may provide an AR/VR device, and the AR/VR device may be any AR/VR device in the AR/VR device group. Optionally, in this embodiment, the above-mentioned AR/VR device can also be replaced by a terminal device such as a mobile terminal.

Optionally, in this embodiment, the above-mentioned AR/VR device may be located in at least one network device among multiple network devices of the computer network.

In this embodiment, the above-mentioned AR/VR device can execute the program code of the following steps in the virtual clothing matching method: in response to the input instructions acting on the operation interface, display clothing images of at least two types of virtual clothing on the operation interface. ; In response to a clothing matching instruction acting on the operating interface, display matching images of at least two types of virtual clothing on the operating interface, where the matching images are used to represent the results obtained by matching at least two types of virtual clothing to virtual objects. The matching image is an image generated by using an image generation model corresponding to at least two types of virtual clothing based on clothing text corresponding to at least two types of virtual clothing, and the clothing text is used to describe at least two types of virtual clothing.

Optionally, FIG. 16 is a structural block diagram of a computer terminal according to an embodiment of the present application. As shown in Figure 16, the computer terminal A may include: one or more (only one is shown in the figure) processors 1602, a memory 1604, a storage controller, and a peripheral interface, wherein the peripheral interface is connected with a radio frequency module, Audio module and monitor connections.

The memory can be used to store software programs and modules, such as the program instructions/modules corresponding to the virtual clothing matching method and device in the embodiments of the present application. The processor executes various tasks by running the software programs and modules stored in the memory. Function application and data processing are to implement the above-mentioned virtual clothing matching method. Memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory may further include memory located remotely relative to the processor, and these remote memories may be connected to terminal A through a network. Examples of the above-mentioned networks include but are not limited to the Internet, intranets, local area networks, mobile communication networks and combinations thereof.

The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: in response to the input instructions acting on the operation interface, display clothing images of at least two types of virtual clothing on the operation interface; in response to the input instructions acting on the operation interface The clothing matching instruction on the operation interface displays matching images of at least two types of virtual clothing on the operation interface, where the matching image is used to represent an image obtained by matching at least two types of virtual clothing to a virtual object, and the matching image It is an image generated by using an image generation model corresponding to at least two types of virtual clothing based on clothing text corresponding to at least two types of virtual clothing, and the clothing text is used to describe at least two types of virtual clothing.

Optionally, the above processor can also execute the program code of the following steps: using the feature extraction module in the image generation model to extract features from the clothing text to obtain clothing features; using the generation module in the image generation model to extract features based on clothing features and object features. ,generate matching images where object features are ,used to characterize virtual objects.

Optionally, the object text used to describe the virtual object is also displayed on the operation interface. The above-mentioned processor can also execute the program code of the following steps: using the feature extraction module to extract features from the object text to obtain object features.

Optionally, the operation interface also displays environmental text describing the virtual environment in which the virtual object is located. The above-mentioned processor can also execute the program code of the following steps: use the feature extraction module to extract features from the environmental text to obtain environmental features. ; Use the generation module to generate matching images based on clothing features, object features and environmental features.

Optionally, the image generation model includes: a trained generation model and a large model. The large model includes at least one first module connected in sequence. The trained generation model includes at least one second module. The second module is related to the first module. For matching, the above-mentioned processor can also execute the program code of the following steps: using the current first module in at least one first module to generate the first matching data based on the clothing text; using the current second module in the at least one second module to generate the first matching data based on the clothing text. Text, generate second matching data, where the current second module is the second module corresponding to the current first module; merge the first matching data and the second matching data to obtain merged data, where the current first module If it is the last module in at least one first module, the merged data is a matching image; if the current first module is not the last module in at least one first module, the merged data is input to at least one a next first module among the first modules and a next second module among at least one second module.

Optionally, the above-mentioned processor can also execute the program code of the following steps: construct a training image containing at least two types of virtual clothing; generate training text corresponding to the training image, wherein the training text is used to classify at least two types of virtual clothing. The clothing is described; the initial generation model is trained using training images and training text to obtain a well-trained generation model.

Optionally, the above-mentioned processor can also execute the program code of the following steps: obtain an original image, wherein the display content of the original image includes at least two types of virtual clothing and at least one object; Segment to obtain the segmented image; superimpose the segmented image to the preset image to generate a training image.

Optionally, the above-mentioned processor can also execute the program code of the following steps: display at least one image set in the operation interface, wherein different image sets contain at least two types of virtual clothing that are different; respond to the at least one image set. Select the instruction, obtain the image collection corresponding to the selection instruction, and obtain the original image.

Optionally, the above processor can also execute the program code of the following steps: using the image text reasoning model to perform text prediction on the training image and generate training text.

The processor can call the information stored in the memory and the application program through the transmission device to perform the following steps: obtain clothing images of at least two types of virtual clothing; determine image generation models and clothing texts corresponding to at least two types of virtual clothing, Among them, the clothing text is used to describe at least two types of virtual clothing; an image generation model is used to generate matching images of at least two types of virtual clothing based on the clothing text, where the matching image is used to represent the combination of at least two types of virtual clothing. An image obtained by matching virtual clothing to a virtual object.

The processor can call the information stored in the memory and the application program through the transmission device to perform the following steps: construct a training image containing at least two types of virtual clothing; generate training text corresponding to the training image, wherein the training text is used to at least Two types of virtual clothing are described; the initial generation model is trained using training images and training text to obtain an image generation model, wherein the image generation model is used to generate at least two types of clothing text corresponding to at least two types of virtual clothing. Matching images of two types of virtual clothing, the clothing text is used to describe at least two types of virtual clothing, and the matching image is used to represent an image obtained by matching at least two types of virtual clothing to a virtual object.

The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: display clothing images of at least two types of virtual clothing on the presentation screen of the virtual reality VR device or the augmented reality AR device; determine at least two types of virtual clothing. The image generation model and clothing text corresponding to each type of virtual clothing, where the clothing text is used to describe at least two types of virtual clothing; the image generation model is used to generate matching images of at least two types of virtual clothing based on the clothing text. , wherein the matching image is used to represent an image obtained by matching at least two types of virtual clothing to a virtual object; driving the VR device or AR device to render and display the matching image.

The processor can call the information stored in the memory and the application program through the transmission device to perform the following steps: call the first interface to obtain clothing images of at least two types of virtual clothing, where the first interface includes a first parameter, and the first parameter The parameter value is a clothing image; determine the image generation model and clothing text corresponding to at least two types of virtual clothing, where the clothing text is used to describe at least two types of virtual clothing; use the image generation model based on the clothing text to generate Matching images of at least two types of virtual clothing, where the matching image is used to represent an image obtained by matching at least two types of virtual clothing to a virtual object; the matching image is output in the operation interface by calling the second interface, where, The second interface includes a second parameter, and the parameter value of the second parameter is the matching image.

Using the embodiments of the present application, a virtual clothing matching solution is provided. Since the matching image is an image obtained by matching at least two types of virtual clothing to a virtual object, the purpose of simulating specific styles of clothing and matching and displaying different types of virtual clothing is achieved, and because the matching image is made using The image generation model is based on images generated by clothing text, and achieves the purpose of using text to generate images to directly generate model pictures with higher authenticity and more image elements, thereby improving the virtual clothing matching effect and expanding the capabilities of virtual clothing matching. The technical effects of the application scenarios thus solve the technical problems in related technologies that the effect of virtual clothing matching through virtual fitting technology is poor and the application scenarios are limited.

Those of ordinary skill in the art can understand that the structure shown in Figure 16 is only illustrative, and the computer terminal can also be a smart phone (such as an Android phone, iOS phone, etc.), a tablet computer, a handheld computer, and a mobile Internet device (Mobile Internet Devices, MID) , PAD and other terminal equipment. FIG. 16 does not limit the structure of the above electronic device. For example, the computer terminal A may also include more or fewer components (such as network interfaces, display devices, etc.) than shown in FIG. 16 , or have a different configuration than that shown in FIG. 16 .

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program. The program can be stored in a computer-readable storage medium, and the storage medium can Including: flash disk, read-only memory (Read-Only Memory, ROM), random access memory (RandomAccess Memory, RAM), magnetic disk or optical disk, etc.

Example 12

Embodiments of the present application also provide a computer-readable storage medium. Optionally, in this embodiment, the above-mentioned computer-readable storage medium can be used to save the program code executed by the virtual clothing matching method provided in the above-mentioned Embodiment 1.

Optionally, in this embodiment, the above computer-readable storage medium can be located in any computer terminal in the AR/VR device terminal group in the AR/VR device network, or in any mobile terminal in the mobile terminal group. .

Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for performing the following steps: displaying at least two types of virtual clothing on the operating interface in response to input instructions acting on the operating interface clothing images; in response to clothing matching instructions acting on the operating interface, display matching images of at least two types of virtual clothing on the operating interface, wherein the matching images are used to represent matching at least two types of virtual clothing to virtual objects The obtained image, the matching image is an image generated by using an image generation model corresponding to at least two types of virtual clothing based on clothing text corresponding to at least two types of virtual clothing, and the clothing text is used to perform at least two types of virtual clothing. describe.

Optionally, the above computer-readable storage medium is also configured to store program codes for performing the following steps: using the feature extraction module in the image generation model to extract features from the clothing text to obtain clothing features; using the feature extraction module in the image generation model. The generation module generates matching images based on clothing features and object features, where the object features are used to characterize virtual objects.

Optionally, the object text used to describe the virtual object is also displayed on the operation interface. The above-mentioned computer-readable storage medium is also configured to store program codes for performing the following steps: using a feature extraction module to extract features from the object text. , get the object characteristics.

Optionally, the operating interface also displays environment text describing the virtual environment in which the virtual object is located. The above computer-readable storage medium is also configured to store program code for performing the following steps: using the feature extraction module to analyze the environment. Features are extracted from the text to obtain environmental features; the generation module is used to generate matching images based on clothing features, object features and environmental features.

Optionally, the image generation model includes: a trained generation model and a large model. The large model includes at least one first module connected in sequence. The trained generation model includes at least one second module. The second module is related to the first module. Matching, the above computer-readable storage medium is further configured to store program code for performing the following steps: using the current first module in at least one first module to generate first matching data based on the clothing text; using at least one second module The current second module in generates second matching data based on the clothing text, where the current second module is the second module corresponding to the current first module; the first matching data and the second matching data are merged to obtain the merged data , wherein, when the current first module is the last module among at least one first module, the merged data is a matching image; when the current first module is not the last module among at least one first module, The merged data is respectively input to a next first module of at least one first module and a next second module of at least one second module.

Optionally, the above computer-readable storage medium is also configured to store program code for performing the following steps: constructing a training image containing at least two types of virtual clothing; generating training text corresponding to the training image, wherein the training text is It is used to describe at least two types of virtual clothing; use training images and training text to train the initial generation model to obtain a well-trained generation model.

Optionally, the above computer-readable storage medium is also configured to store program codes for performing the following steps: obtaining an original image, wherein the display content of the original image includes at least two types of virtual clothing and at least one object; The area where at least one object in the image is located is segmented to obtain a segmented image; the segmented image is superimposed on the preset image to generate a training image.

Optionally, the above-mentioned computer-readable storage medium is further configured to store program codes for performing the following steps: displaying at least one image collection in the operation interface, wherein different image collections contain at least two different types of virtual clothing; In response to a selection instruction acting on at least one image set, the image set corresponding to the selection instruction is obtained, and the original image is obtained.

Optionally, the above computer-readable storage medium is further configured to store program codes for performing the following steps: using an image-text reasoning model to perform text prediction on training images and generate training text.

Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for performing the following steps: obtaining clothing images of at least two types of virtual clothing; determining the clothing images corresponding to the at least two types of virtual clothing. Image generation model and clothing text, wherein the clothing text is used to describe at least two types of virtual clothing; using the image generation model to generate matching images of at least two types of virtual clothing based on the clothing text, wherein the matching image is used to Characterizes an image obtained by matching at least two types of virtual clothing to a virtual object.

Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: constructing a training image containing at least two types of virtual clothing; generating training text corresponding to the training image, wherein , the training text is used to describe at least two types of virtual clothing; the training image and the training text are used to train the initial generation model to obtain an image generation model, in which the image generation model is used to correspond to at least two types of virtual clothing. The clothing text is used to generate matching images of at least two types of virtual clothing. The clothing text is used to describe at least two types of virtual clothing. The matching image is used to represent the results obtained by matching at least two types of virtual clothing to virtual objects. Image.

Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for performing the following steps: displaying at least two types of virtual clothing on the presentation screen of the virtual reality VR device or the augmented reality AR device clothing images; determine the image generation models and clothing texts corresponding to at least two types of virtual clothing, where the clothing text is used to describe at least two types of virtual clothing; use the image generation model to generate at least two types of clothing based on the clothing text. Matching images of types of virtual clothing, where the matching images are used to represent images obtained by matching at least two types of virtual clothing to virtual objects; driving the VR device or AR device to render and display the matching images.

Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for performing the following steps: calling a first interface to obtain clothing images of at least two types of virtual clothing, wherein the first interface includes The first parameter, the parameter value of the first parameter is a clothing image; determine the image generation model and clothing text corresponding to at least two types of virtual clothing, where the clothing text is used to describe at least two types of virtual clothing; use the image The generation model generates matching images of at least two types of virtual clothing based on the clothing text, where the matching images are used to represent images obtained by matching at least two types of virtual clothing to virtual objects; by calling the second interface, in the operation interface The matching image is output, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the matching image.

The above serial numbers of the embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

In the above-mentioned embodiments of the present application, each embodiment is described with its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or may be Integrated into another system, or some features can be ignored, or not implemented. On the other hand, the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the units or modules may be in electrical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application can be integrated into one processing unit, each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated units can be implemented in the form of hardware or software functional units.

If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium , including several instructions to cause a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The aforementioned storage media include: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program code. .

The above are only the preferred embodiments of the present application. It should be pointed out that for those of ordinary skill in the art, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications can also be made. should be regarded as the scope of protection of this application.

Claims (14)

1. A virtual clothing matching method, characterized by including: In response to an input instruction acting on the operation interface, display clothing images of at least two types of virtual clothing on the operation interface; In response to a clothing matching instruction acting on the operation interface, matching images of the at least two types of virtual clothing are displayed on the operation interface, wherein the matching images are used to represent the combination of the at least two types of virtual clothing. An image obtained by matching virtual clothing to a virtual object, where the matching image is an image generated based on the clothing text corresponding to the at least two types of virtual clothing using an image generation model corresponding to the at least two types of virtual clothing, so The clothing text is used to describe the at least two types of virtual clothing. 2. The method according to claim 1, characterized in that, the method further comprises: Use the feature extraction module in the image generation model to extract features from the clothing text to obtain clothing features; The matching image is generated based on the clothing features and object features using a generation module in the image generation model, where the object features are used to characterize the virtual object. 3. The method according to claim 2, wherein object text for describing the virtual object is also displayed on the operation interface, and the method further includes: The feature extraction module is used to extract features from the object text to obtain the object features. 4. The method according to claim 2, characterized in that the operation interface also displays environmental text for describing the virtual environment in which the virtual object is located, and the generation module in the image generation model is used based on The clothing characteristics and object characteristics are used to generate the matching image, including: Use the feature extraction module to extract features from the environmental text to obtain environmental features; The matching image is generated based on the clothing features, the object features and the environment features using the generating module. 5. The method according to claim 1, characterized in that the image generation model includes: a trained generation model and a large model, the large model includes at least one first module connected in sequence, the trained The generated model includes at least one second module that matches the first module, and the method further includes: Utilize the current first module in the at least one first module to generate first matching data based on the clothing text; Utilize the current second module in the at least one second module to generate second matching data based on the clothing text, wherein the current second module is the second module corresponding to the current first module; The first matching data and the second matching data are merged to obtain merged data, wherein, when the current first module is the last module among the at least one first module, the merge The data is the matching image; In the case where the current first module is not the last module among the at least one first module, the merged data is respectively input to the next first module among the at least one first module and the next first module among the at least one first module. The next second module of at least one second module. 6. The method according to claim 5, characterized in that, the method further comprises: Constructing training images containing the at least two types of virtual clothing; Generate training text corresponding to the training image, wherein the training text is used to describe the at least two types of virtual clothing; The initial generation model is trained using the training image and the training text to obtain the trained generation model. 7. The method of claim 6, wherein constructing a training image containing the at least two types of virtual clothing includes: Obtaining an original image, wherein the display content of the original image includes the at least two types of virtual clothing and at least one object; Segment the area where the at least one object is located in the original image to obtain a segmented image; The segmented image is superimposed on a preset image to generate the training image. 8. The method of claim 7, wherein obtaining the original image includes: Display at least one image set in the operation interface, wherein the at least two types of virtual clothing contained in different image sets are different; In response to a selection instruction acting on the at least one image set, the image set corresponding to the selection instruction is obtained to obtain the original image. 9. A virtual clothing matching method, characterized by including: Obtain clothing images of at least two types of virtual clothing; Determine the image generation model and clothing text corresponding to the at least two types of virtual clothing, wherein the clothing text is used to describe the at least two types of virtual clothing; The image generation model is used to generate matching images of the at least two types of virtual clothing based on the clothing text, wherein the matching images are used to represent the matching of the at least two types of virtual clothing to virtual objects. the resulting image. 10. A training method for an image generation model, characterized by comprising: Constructing training images containing at least two types of virtual clothing; Generate training text corresponding to the training image, wherein the training text is used to describe the at least two types of virtual clothing; The initial generation model is trained using the training image and the training text to obtain an image generation model, wherein the image generation model is used to generate the at least two types of virtual clothing based on clothing text corresponding to the at least two types of virtual clothing. Matching images of two types of virtual clothing, the clothing text is used to describe the at least two types of virtual clothing, and the matching image is used to represent matching the at least two types of virtual clothing to a virtual object the resulting image. 11. A virtual clothing matching method, characterized by including: Display clothing images of at least two types of virtual clothing on the presentation screen of the virtual reality VR device or the augmented reality AR device; Determine the image generation model and clothing text corresponding to the at least two types of virtual clothing, wherein the clothing text is used to describe the at least two types of virtual clothing; The image generation model is used to generate matching images of the at least two types of virtual clothing based on the clothing text, wherein the matching images are used to represent the matching of the at least two types of virtual clothing to virtual objects. the resulting image; Driving the VR device or the AR device to render and display the matching image. 12. A virtual clothing matching method, characterized by including: Obtain clothing images of at least two types of virtual clothing by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the clothing image; Determine the image generation model and clothing text corresponding to the at least two types of virtual clothing, wherein the clothing text is used to describe the at least two types of virtual clothing; The image generation model is used to generate matching images of the at least two types of virtual clothing based on the clothing text, wherein the matching images are used to represent the matching of the at least two types of virtual clothing to virtual objects. the resulting image; The matching image is output in the operation interface by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the matching image. 13. An electronic device, characterized in that it includes: Memory, which stores executable programs; A processor, configured to run the program, wherein the method according to any one of claims 1 to 12 is executed when the program is run. 14. A computer-readable storage medium, characterized in that the computer-readable storage medium includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the right The method according to any one of claims 1 to 12.