CN115169577A - Model training method and data processing method - Google Patents

Abstract

The application discloses a model training method and a data processing method. The model training method comprises the following steps: obtaining training samples, the training samples comprising: different types of first training data and second training data; processing the first training data by using a first processing model to obtain a first global feature and a first data block feature, wherein the first global feature is used for representing the semantic feature of the first training data, and the first data block feature is used for representing the feature of a data block in the first training data; processing the second training data by using a second processing model to obtain a second global feature and a second data block feature; and adjusting parameters of the first processing model and the second processing model based on the global similarity of the first global feature and the second global feature and the data block similarity of the first data block feature and the second data block feature. The method and the device solve the technical problem that the accuracy of processing the multi-modal data through the model is poor in the related technology.

Description

Model training method and data processing method

technical field

The present application relates to the field of data processing, and in particular, to a model training method and a data processing method.

Background technique

The information on the Internet is mainly video, image, and text, and the user's search request is mainly text. However, the feature vectors extracted by the above models cannot accurately represent the detailed information of images and texts, resulting in poor model processing accuracy.

For the above problems, no effective solution has been proposed yet.

SUMMARY OF THE INVENTION

Embodiments of the present application provide a model training method and a data processing method, so as to at least solve the technical problem in the related art that the accuracy of processing multimodal data through a model is poor.

According to an aspect of the embodiments of the present application, a model training method is provided, including: acquiring training samples, wherein the training samples include: different types of first training data and second training data; The training data is processed to obtain a first global feature and a first data block feature, wherein the first global feature is used to represent the semantic feature of the first training data, and the first data block feature is used to represent the data block in the first training data feature; use the second processing model to process the second training data to obtain the second global feature and the second data block feature; based on the global similarity of the first global feature and the second global feature, the first data block feature and the first data block feature The data block similarity of the second data block feature adjusts the parameters of the first processing model and the second processing model, and the first processing model and the second processing model are machine learning models.

According to another aspect of the embodiments of the present application, a data processing method is also provided, including: acquiring different types of first target data and multiple second target data; processing the first target data by using a first processing model, Obtain the first feature vector corresponding to the first target data; use the second processing model to process the plurality of second target data respectively to obtain the second feature vector corresponding to the plurality of second target data; The feature vector is matched to determine the push data in the plurality of second target data, wherein the push data is used to represent the second target data that matches the first target data; wherein, the parameters of the first processing model and the second processing model The adjustment is made based on the global similarity of the first global feature and the second global feature, and the data block similarity of the first data block feature and the second data block feature, and the first global feature and the first data block feature are adjusted by the first processing model. The first training data is processed and obtained, the second global feature and the second data block feature are obtained by processing the second training data through the second processing model, the first training data and the second training data are different types of data, the first global feature The feature is used to represent the semantic feature of the first training data, the first data block feature is used to represent the feature of the data block in the first training data, and the first processing model and the second processing model are machine learning models.

According to another aspect of the embodiments of the present application, a data processing method is also provided, including: displaying first target data on the operation interface in response to an input instruction acting on the operation interface of the client; A push instruction, displaying the push data in the plurality of second target data on the operation interface, wherein the push data is used to represent the second target data matching the first target data, the first target data and the plurality of second target data The push data is obtained by matching the first feature vector corresponding to the first target data with the second feature vectors corresponding to multiple second target data, and the first feature vector is processed by the first processing model on the first target data. The second feature vector is obtained by processing the second target data through the second processing model, and the parameters of the first processing model and the second processing model are based on the global similarity of the first global feature and the second global feature, the first data The data block similarity between the block feature and the second data block feature is adjusted. The first global feature and the first data block feature are obtained by processing the first training data through the first processing model, and the second global feature and the second data block feature are obtained. The second training data is processed by the second processing model. The first training data and the second training data are different types of data. The first global feature is used to represent the semantic feature of the first training data, and the first data block feature is used for For characterizing the features of the data blocks in the first training data, the first processing model and the second processing model are machine learning models.

According to another aspect of the embodiments of the present application, a data processing method is also provided, including: displaying first target data on a presentation screen of a virtual reality VR device or an augmented reality AR device; acquiring a plurality of second target data, wherein , the types of the first target data and the plurality of second target data are different; use the first processing model to process the first target data to obtain the first feature vector corresponding to the first target data; use the second processing model to process the multiple The second target data is processed to obtain a plurality of second feature vectors corresponding to the second target data; the first feature vector and the second feature vector are matched to determine the push data in the plurality of second target data, wherein the push data Used to characterize the second target data that matches the first target data; drive the VR device or AR device to display the push data; wherein, the parameters of the first processing model and the second processing model are based on the first global feature and the second global feature. The global similarity, the data block similarity of the first data block feature and the second data block feature are adjusted. The first global feature and the first data block feature are obtained by processing the first training data by the first processing model. The feature and the second data block feature are obtained by processing the second training data by the second processing model. The first training data and the second training data are different types of data, and the first global feature is used to represent the semantic feature of the first training data. , the first data block feature is used to represent the feature of the data block in the first training data, and the first processing model and the second processing model are machine learning models.

According to another aspect of the embodiments of the present application, a model training method is also provided, including: the server obtains a model training request sent by the client by invoking a first interface, wherein the first interface includes a first parameter, and the value of the first parameter The parameter value is a model training request, and the model training request is used to train the first processing model and the second processing model; the server obtains training samples based on the model training request, wherein the training samples include: different types of first training data and second training data; the server processes the first training data by using the first processing model to obtain a first global feature and a first data block feature, wherein the first global feature is used to represent the semantic feature of the first training data, and the first data block The feature is used to characterize the feature of the data block in the first training data; the server uses the second processing model to process the second training data to obtain the second global feature and the second data block feature; the server is based on the first global feature and the second The global similarity of the global feature, the data block similarity of the first data block feature and the second data block feature, adjust the parameters of the first processing model and the second processing model; the server outputs the first processing model by calling the second interface and a second processing model to the client, wherein the second interface includes a second parameter, the parameter values of the second parameter are the first processing model and the second processing model, and the first processing model and the second processing model are machine learning models.

According to another aspect of the embodiments of the present application, a data processing method is further provided, including: the server obtains the first target data sent by the client by calling a first interface, wherein the first interface includes a first parameter, and the first parameter The parameter value is the first target data; the server obtains a plurality of second target data, wherein the first target data and the plurality of second target data are of different types; the server processes the first target data by using the first processing model, and obtains The first feature vector corresponding to the first target data; the server uses the second processing model to process the plurality of second target data respectively to obtain the second feature vector corresponding to the plurality of second target data; the server combines the first feature vector with the first feature vector. The two feature vectors are matched to determine the push data in the plurality of second target data, wherein the push data is used to represent the second target data that matches the first target data; the server outputs the push data to the client by calling the second interface , wherein the second interface includes a second parameter, and the parameter value of the second parameter is the push data; wherein the parameters of the first processing model and the second processing model are based on the global similarity of the first global feature and the second global feature, the first The data block similarity between a data block feature and a second data block feature is adjusted. The first global feature and the first data block feature are obtained by processing the first training data through the first processing model. The second global feature and the second data The block feature is obtained by processing the second training data by the second processing model, the first training data and the second training data are different types of data, the first global feature is used to represent the semantic feature of the first training data, and the first data block The feature is used to characterize the feature of the data block in the first training data, and the first processing model and the second processing model are machine learning models.

According to another aspect of the embodiments of the present application, a computer-readable storage medium is also provided, where the computer-readable storage medium includes a stored program, wherein, when the program runs, the device where the computer-readable storage medium is located is controlled to perform the above-mentioned embodiments. any method.

According to another aspect of the embodiments of the present application, a computer terminal is also provided, including: a processor; a memory connected to the processor and configured to provide the processor with instructions for processing the following processing steps: acquiring training samples, wherein, The training samples include: different types of first training data and second training data; using the first processing model to process the first training data to obtain a first global feature and a first data block feature, wherein the first global feature is used for Characterizing the semantic features of the first training data, and the first data block features are used to characterize the features of the data blocks in the first training data; using the second processing model to process the second training data to obtain second global features and second data block feature; based on the global similarity of the first global feature and the second global feature, the data block similarity of the first data block feature and the second data block feature, the parameters of the first processing model and the second processing model are adjusted, The first processing model and the second processing model are machine learning models.

In the embodiment of the present application, after obtaining the first training data and the second training data of different types, the first training data may be processed by using the first processing model to obtain the first global feature and the first data block feature, And use the second processing model to process the second training data to obtain the second global feature and the second data block feature, and finally based on the global similarity of the first global feature and the second global feature, the first data block feature and the second data block feature. The data block similarity of the data block features adjusts the parameters of the first processing model and the second processing model, so as to achieve the purpose of training the feature extraction model. It is easy to notice that since the adjustment of model parameters is based on the global similarity and the similarity of data blocks, it can achieve a more accurate expression of the similarity between images and texts, improve the training effect of the model, and further improve the model’s ability to adapt to multiple models. The technical effect of the accuracy of processing modal data is solved, thereby solving the technical problem of poor accuracy of processing multi-modal data through a model in the related art. In the image and text search scenario, the first processing model and the second processing model can extract image features and text features with higher accuracy, further making the searched information more accurate and more in line with the user's search needs, thus achieving The effect of improving the accuracy of image and text search.

Description of drawings

The drawings described herein are used to provide further understanding of the present application and constitute a part of the present application. The schematic embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation of the present application. In the attached image:

1 is a block diagram of a hardware structure of a computer terminal (or mobile device) for implementing a model training method according to an embodiment of the present application;

2 is a structural block diagram of a computing environment of a model training method according to an embodiment of the present application;

3 is a flowchart of a model training method according to an embodiment of the present application;

4 is a flowchart of an optional model training method according to an embodiment of the present application;

5 is a flowchart of a first data processing method according to an embodiment of the present application;

6 is a flowchart of a second data processing method according to an embodiment of the present application;

7 is a flowchart of a third data processing method according to an embodiment of the present application;

8 is a flowchart of another model training method according to an embodiment of the present application;

9 is a flowchart of a fourth data processing method according to an embodiment of the present application;

10 is a schematic diagram of a model training apparatus according to an embodiment of the present application;

11 is a schematic diagram of a first data processing apparatus according to an embodiment of the present application;

12 is a schematic diagram of a second data processing apparatus according to an embodiment of the present application;

13 is a schematic diagram of a third data processing apparatus according to an embodiment of the present application;

14 is a schematic diagram of another model training apparatus according to an embodiment of the present application;

15 is a schematic diagram of a fourth data processing apparatus according to an embodiment of the present application;

FIG. 16 is a structural block diagram of a computer terminal according to an embodiment of the present application.

Detailed ways

In order to make those skilled in the art better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only The embodiments are part of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work shall fall within the scope of protection of the present application.

It should be noted that the terms "first", "second", etc. in the description and claims of the present application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific sequence or sequence. It is to be understood that data so used may be interchanged under appropriate circumstances so that the embodiments of the application described herein can be practiced in sequences other than those illustrated or described herein. Furthermore, the terms "comprising" and "having", and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those expressly listed Rather, those steps or units may include other steps or units not expressly listed or inherent to these processes, methods, products or devices.

First of all, some nouns or terms that appear in the process of describing the embodiments of the present application are suitable for the following explanations:

Multi-modal feature learning: Through machine learning technology, a feature encoding model can be trained, which can map the image or text data input by the user into a feature vector, and the model can be used for tasks such as image and text search.

Transformer Model: A neural network model that can perform feature extraction on input images or text. Before inputting an image, the image needs to be divided into multiple image blocks (for example, divided into 16*16 small blocks that do not overlap each other), and each image block is an image token. Similarly, before entering text, the text needs to be divided into multiple text tokens, and each text token can be approximately understood as a word.

InfoNCE (Info Noise-contrastive estimation) loss function: The core idea is to make the similarity between positive sample pairs greater than the similarity between negative sample pairs.

At present, in multimodal feature learning, the CLIP (Contrastive Language-Image Pre-training) model can be used. After inputting image or text data, the image or text can be processed by neural networks such as CNN or Transformer. The data is represented as a global feature vector. In the training results, first extract feature vectors for all images and texts, and then require that the similarity between the feature vectors of the positive sample image-text pair is greater than the similarity between the feature vectors of the negative sample image-text pair.

However, the model compresses image or text data into a global feature vector by averaging or other methods, which will lose more image or text detail data, resulting in inability to distinguish positive samples from difficult negative samples in the training stage. Then, noise is introduced in the training process, which affects the model effect.

In order to solve the above problems, the present application provides an implementation scheme. By establishing a many-to-many correspondence between image tokens and text tokens, the similarity between images or text data can be more accurately expressed, thereby improving the effect of feature training. , to further improve the accuracy of model processing, so that users can be provided with more accurate retrieval results in image and text search tasks.

Example 1

According to an embodiment of the present application, a model training method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings may be executed in a computer system such as a set of computer-executable instructions, and although the flowchart A logical order is shown in the figures, but in some cases steps shown or described may be performed in an order different from that herein.

The method embodiments provided in the embodiments of the present application may be executed in a mobile terminal, a computer terminal, or a similar computing device. FIG. 1 shows a block diagram of the hardware structure of a computer terminal (or mobile device) for implementing the model training method. As shown in FIG. 1, the computer terminal 10 (or the mobile device 10) may include one or more processors 102 (represented by 102a, 102b, . processing means such as a processor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission means 106 for communication functions. In addition, may also include: display, input/output interface (I/O interface), universal serial bus (USB) port (may be included as one of the ports of the BUS bus), network interface, power supply and/or or camera. Those of ordinary skill in the art can understand that the structure shown in FIG. 1 is only a schematic diagram, which does not limit the structure of the above electronic device. For example, the computer terminal 10 may also include more or fewer components than shown in FIG. 1 , or have a different configuration than that shown in FIG. 1 .

It should be noted that the one or more processors 102 and/or other data processing circuits described above may generally be referred to herein as "data processing circuits." The data processing circuit may be embodied in whole or in part as software, hardware, firmware or any other combination. Furthermore, the data processing circuitry may be a single stand-alone processing module, or incorporated in whole or in part into any of the other elements in the computer terminal 10 (or mobile device). The data processing circuit acts as a kind of processor control (eg selection of variable resistor termination paths connected to the interface).

The memory 104 can be used to store software programs and modules of the application software, such as program instructions/data storage devices corresponding to the model training method in the embodiment of the present application. The processor 102 executes the software programs and modules stored in the memory 104 by running Various functional applications and data processing implement the above-mentioned model training method. Memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, memory 104 may further include memory located remotely from processor 102, which may be connected to computer terminal 10 through a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

Transmission means 106 are used to receive or transmit data via a network. A specific example of the above-mentioned network may include a wireless network provided by a communication provider of the computer terminal 10 . In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (Radio Frequency, RF) module, which is used for wirelessly communicating with the Internet.

The display may be, for example, a touch screen type liquid crystal display (LCD) that enables a user to interact with the user interface of the computer terminal 10 (or mobile device).

The block diagram of the hardware structure shown in FIG. 1 can be used not only as an exemplary block diagram of the above-mentioned computer terminal (or mobile device), but also as an exemplary block diagram of the above-mentioned server. In an optional embodiment, FIG. 2 shows a block diagram as a block diagram. An embodiment of using the computer terminal (or mobile device) shown in FIG. 1 above as a computing node in the computing environment 201 is presented. FIG. 2 is a structural block diagram of a computing environment of a model training method according to an embodiment of the present application. As shown in FIG. 2, the computing environment 201 includes a plurality of (210-1, 210- 2, ..., to show) computing nodes (such as servers). Each computing node contains local processing and memory resources, and end users 202 can run applications or store data remotely in computing environment 201 . Applications may be provided as a plurality of services 220-1, 220-2, 220-3 and 220-4 in computing environment 301, representing services "A", "D", "E" and "H", respectively.

End user 202 may provide and access services through a web browser or other software application on a client, and in some embodiments, end user 202 offers and/or requests may be provided to portal gateway 230 . Ingress gateway 230 may include a corresponding proxy to handle offers and/or requests for service 220 (one or more services provided in computing environment 201).

Services 220 are provided or deployed according to various virtualization technologies supported by computing environment 201 . In some embodiments, service 220 may be provided according to virtual machine (VM) based virtualization, container based virtualization, and/or the like. Virtual machine-based virtualization can emulate a real computer by initializing a virtual machine, executing programs and applications without directly touching any actual hardware resources. While virtual machines virtualize machines, according to container-based virtualization, containers can be launched to virtualize an entire operating system (OS) so that multiple workloads can run on a single OS instance.

In one embodiment based on container virtualization, several containers of service 220 may be assembled into a POD (eg, a Kubernetes POD). For example, as shown in FIG. 2, a service 220-2 may be equipped with one or more PODs 240-1, 240-2, . . . , 240-N (collectively referred to as PODs 240). Each POD 240 may include an agent 245 and one or more containers 242-1, 242-2, . . . , 242-M (collectively, containers 242). One or more containers 242 in the POD 240 handle requests related to one or more corresponding functions of the service, and the proxy 245 typically controls network functions related to the service, such as routing, load balancing, and the like. Other services 220 may also accompany PODs similar to PODs 240 .

During operation, execution of a user request from end user 202 may require invocation of one or more services 220 in computing environment 201 , execution of one or more functions of one service 220 may require invocation of one or more of another service 220 Function. As shown in FIG. 2, service "A" 220-1 receives a user request from end user 202 from portal gateway 230, service "A" 220-1 may call service "D" 220-2, and service "D" 220-2 may Service "E" 220-3 is requested to perform one or more functions.

The above-mentioned computing environment may be a cloud computing environment, and the allocation of resources is managed by the cloud service provider, allowing the development of functions without considering the implementation, adjustment or expansion of the server. This computing environment allows developers to execute code in response to events without building or maintaining complex infrastructure. Instead of scaling a single hardware device to handle the underlying load, services can be split up to perform a set of functions that can be scaled independently and automatically.

Under the above operating environment, the present application provides a model training method as shown in FIG. 3 . It should be noted that the model training method in this embodiment may be executed by the computer terminal of the embodiment shown in FIG. 1 . FIG. 3 is a flowchart of a model training method according to an embodiment of the present application. As shown in Figure 3, the method may include the following steps:

Step S302, acquiring training samples, wherein the training samples include: different types of first training data and second training data.

The training samples in the above steps may be multi-modal data, that is, the training samples may contain two different types of data, namely the first training data and the second training data, and the specific data types may be determined according to the application scenario, In the embodiment of the present application, taking the image and text search scenario as an example, the first training data may be image data, and the second training data may be text data, but not limited thereto.

In addition, in order to improve the model effect, the training samples can be divided into positive samples and negative samples, and the positive samples contain two different types of data that match. Spinning dress", the image data can be an image of "chiffon dress"; for negative samples, text data is "chiffon dress", and image data can be an image of "star A".

In an optional embodiment, the purpose of training models for different users is often different. In order to ensure that the trained model can meet the needs of users, the multi-modal data provided by the user or the multi-modal data stored in the user server can be used. as a training sample. In another optional embodiment, the number of training samples provided by the user is limited. In order to further improve the accuracy of model processing, a large amount of multimodal data may be obtained from the Internet as training samples based on user requirements.

Step S304, using the first processing model to process the first training data to obtain a first global feature and a first data block feature, wherein the first global feature is used to represent the semantic feature of the first training data, and the first data block feature A feature used to characterize the data block in the first training data.

The first processing model in the above steps may be a machine learning model capable of extracting features from the first training data, and the model may be obtained by training with the same type of data as the first training data, and the model may be implemented in related technologies. The model structure of feature extraction is described by taking the Transformer model as an example in the embodiments of the present application, but it is not limited to this. In the image-text search scenario, when the first training data is image data, the first processing model may be an image encoder; when the first training data is text data, the first processing model may be a text encoder.

In an optional embodiment, when feature extraction needs to be performed on the first training data, the first training data is first divided to obtain multiple data blocks patchtoken, and then a learnable embedding vector classtoken is added to obtain The input sequence is finally input to the first processing model, wherein the class token can represent the overall semantics of the first training data. After the input sequence is input into the first processing model for feature extraction, the feature vector corresponding to the class token, that is, the aforementioned first global feature, and the feature vector corresponding to each patch token, that is, the aforementioned first data block feature can be obtained.

Step S306, using the second processing model to process the second training data to obtain the second global feature and the second data block feature.

The second processing model in the above steps may be a machine learning model capable of performing feature extraction on the first training data. Since the first training data and the second training data are two different types of data, it is necessary to train the first processing model and the second processing model respectively for the first training data and the second training data, so as to ensure the accuracy of feature extraction . In the image-text search scenario, when the second training data is text data, the second processing model may be a text encoder; when the second training data is image data, the second processing model may be an image encoder.

Since the processing purposes of the first processing model and the second processing model are the same, the structure of the second processing model is the same as that of the first processing model, and the processing flow of the second processing model is the same as the processing flow of the first processing model. I won't go into details.

Step S308: Adjust the parameters of the first processing model and the second processing model based on the global similarity between the first global feature and the second global feature, and the data block similarity between the second data block feature and the second data block feature.

The global similarity in the above steps can be a one-to-one similarity relationship between the first global feature and the second global feature, and the data block similarity can be a many-to-many relationship between the first data block feature and the second data block feature. similarity relationship.

In an optional embodiment, the training target of the first processing model and the second processing model may be that the similarity of positive samples is greater than the similarity of negative samples, therefore, a loss function may be constructed based on the global similarity and the data block similarity , and further adjust the parameters of the first processing model and the second processing model to ensure that the model accuracy of the two models is the same.

It should be noted that the specific adjustment process of the parameters can be implemented by using the method provided in the related art, which is not specifically limited in this application.

A preferred embodiment of the present application will be described in detail below with reference to FIG. 4 taking a picture and text search scenario as an example. As shown in Figure 4, the training sample contains an image and text, the image has a pizza drawn on it, the content of the text is "sausage pizza plate", the image can be feature extracted by the image encoder, and the text can be feature extracted by the text encoder . Before the image encoder processes the image, the image is first divided into 6 image blocks, that is, 6 image tokens, and then the 6 image tokens and a class token are input to the image encoder together to obtain Image global features and 6 image token features. Before the text encoder processes the text, the text is first divided into 6 words, that is, 6 text tokens, and then the 6 text tokens and another class token are input into the text editor together to get Text global features and 6 text token features. Image global features and text global features can be used to construct global feature alignment loss, and 6 image tokens and 6 text tokens can be processed by the token feature alignment module to construct token feature alignment loss. The parameters of image encoder and text encoder are updated through global feature alignment loss and token feature alignment loss to achieve the purpose of model training. By establishing a one-to-one correspondence between image global features and text global features, as well as a many-to-many correspondence between image tokens and text tokens, the similarity between images and texts can be more accurately expressed. Improve the performance of feature training.

Through the solutions provided by the above embodiments of the present application, after obtaining different types of first training data and second training data, the first processing model can be used to process the first training data to obtain first global features and first data block feature, and use the second processing model to process the second training data to obtain the second global feature and the second data block feature, and finally based on the global similarity of the first global feature and the second global feature, the first data block feature The data block similarity with the second data block feature is used to adjust the parameters of the first processing model and the second processing model, so as to achieve the purpose of training the feature extraction model. It is easy to notice that since the adjustment of model parameters is based on the global similarity and the similarity of data blocks, it can achieve a more accurate expression of the similarity between images and texts, improve the training effect of the model, and further improve the model’s ability to adapt to multiple models. The technical effect of the accuracy of processing modal data is solved, thereby solving the technical problem of poor accuracy of processing multi-modal data through a model in the related art. In the image and text search scenario, the first processing model and the second processing model can extract image features and text features with higher accuracy, further making the searched information more accurate and more in line with the user's search needs, thus achieving The effect of improving the accuracy of image and text search.

In the above embodiments of the present application, based on the global similarity between the first global feature and the second global feature, and the data block similarity between the second data block feature and the second data block feature, the first processing model and the second processing model The parameters are adjusted, including: constructing the first loss function based on the global similarity; constructing the second loss function based on the similarity of the data block; obtaining the weighted sum of the first loss function and the second loss function to obtain the target loss function; based on the target loss The function adjusts the parameters of the first processing model and the second processing model.

The above-mentioned first loss function and second loss function can be set according to the actual training target. For example, in the image and text search scenario, the training target is the similarity between the feature vectors of the positive samples, which is greater than the similarity between the feature vectors of the negative samples. Similarity, therefore, the first loss function and the second loss function may be InfoNCE loss functions, but are not limited thereto.

In an alternative embodiment, the training samples may contain positive samples and negative samples, for example, as shown in Figure 4, "sausage pizza plate" and the image represent the same item, therefore, the image shown in Figure 4 and The text is a positive sample. Therefore, the global similarity contains the first global similarity of positive samples and the second global similarity of negative samples, the data block similarity contains the first data block similarity of positive samples and the second data block similarity of negative samples, and then According to the formula of the InfoNCE loss function, the first loss function and the second loss function are constructed respectively, and then the target loss function is obtained by weighting the function values of the two loss functions. The parameters of the first processing model and the second processing model are adjusted, and the training samples are used to continue training until the target loss function is smaller than the threshold value, stop adjusting the parameters of the first processing model and the second processing model, and determine that the entire training process is over.

In the above-mentioned embodiment of the present application, constructing the first loss function based on the global similarity includes: obtaining the first global similarity corresponding to the first sample and the second global similarity corresponding to the second sample from the global similarity, Wherein, two different types of data contained in the first sample match, and two different types of data contained in the second sample do not match; a first loss function is constructed based on the first global similarity and the second global similarity.

In an optional embodiment, the first loss function L _t can be constructed by the following formula:

和

表示第二全局相似度，τ表示超参数。需要说明的是，上述公式给出了使用两个第二全局相似度构建第一损失函数的实现方案，但不仅限于此，可以根据实际需要确定需要使用的第二全局相似度的数量。Among them, _sp represents the first global similarity,

and

represents the second global similarity, and τ represents the hyperparameter. It should be noted that the above formula provides an implementation scheme of using two second global similarities to construct the first loss function, but it is not limited to this, and the number of the second global similarities to be used can be determined according to actual needs.

In the above-mentioned embodiment of the present application, constructing the second loss function based on the similarity of the data blocks includes: obtaining the similarity of the first data block corresponding to the first sample and the second data corresponding to the second sample from the similarity of the data block Block similarity; builds a second loss function based on the first data block similarity and the second data block similarity.

It should be noted that both the second loss function and the first loss function are InfoNCE loss functions. Therefore, the construction process of the second loss function is the same as that of the first loss function, which is not repeated here.

In the above-mentioned embodiment of the present application, the method further includes: using a feature alignment module to perform feature alignment on the second data block feature and the second data block feature to obtain the data block similarity.

The feature alignment module in the above steps can be any model provided in the related art that can achieve the purpose of token feature alignment. In this embodiment of the present application, the feature alignment module can be implemented through one layer of cross attention.

In an optional embodiment, since the number of divisions of multimodal data in practical applications is large, and token feature alignment needs to process all token features, in order to improve model training efficiency and save labor costs, The feature alignment module provided in the related art can be used to achieve token feature alignment to determine the similarity of data blocks.

In the above-mentioned embodiment of the present application, the method further includes: acquiring first target data; processing the first target data by using a first processing model to obtain a first feature vector corresponding to the first target data; using the second processing model to separate Process a plurality of second target data to obtain second feature vectors corresponding to the plurality of second target data; match the first feature vector and the second feature vector to determine the push data in the plurality of second target data, wherein , the push data is used to represent the second target data that matches the first target data.

In an optional embodiment, in order to achieve the purpose of multi-modal data retrieval and data push, the user can input the first target data that needs to be retrieved or pushed, and use the first process trained by the above method. The model performs feature extraction on the first target data to obtain a first feature vector; in addition, for a plurality of second target data stored in the database (that is, the data that needs to be pushed to the user), the second processing trained by the above method can be used. The model performs feature extraction on the second target data to obtain a plurality of second feature vectors, each of which corresponds to a second target data. Then, by matching the first feature vector with each second feature vector, a second feature vector that can be successfully matched can be determined, and then the second target data corresponding to the second feature vector is used as the retrieval result that is finally pushed to the user or push data.

It should be noted that, for the sake of simple description, the foregoing method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the described action sequence. Because in accordance with the present application, certain steps may be performed in other orders or concurrently. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

From the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is better implementation. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or in a part that contributes to the prior art, and the computer software product is stored in a storage medium (such as ROM/RAM, magnetic disk, CD-ROM), including several instructions to make a terminal device (which may be a mobile phone, a computer, a server, or a network device, etc.) execute the methods described in the various embodiments of this application.

Example 2

According to an embodiment of the present application, a data processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings may be executed in a computer system such as a set of computer-executable instructions, and although the flowchart A logical order is shown in the figures, but in some cases steps shown or described may be performed in an order different from that herein.

FIG. 5 is a flowchart of a first data processing method according to an embodiment of the present application. As shown in Figure 5, the method may include the following steps:

Step S502, acquiring different types of first target data and multiple second target data.

The first target data and the plurality of second target data in the above steps may be multimodal data, that is, they belong to two different types. The second target data may be image data, or the first target data may be image data, and the plurality of second target data may be text data.

In an optional embodiment, the first target data may be provided by a user. For example, taking text data as an example, the first target data may be text manually input by a user. Taking image data as an example, the first target data may be An image captured by the user or filtered by the user from captured images or images from the Internet. The plurality of second target data may be data stored in a database or data in the Internet.

Step S504, using the first processing model to process the first target data to obtain a first feature vector corresponding to the first target data.

The first feature vector in the above steps may be a vector obtained by compressing the input sequence input to the first processing model, where the input sequence may include multiple data blocks obtained by dividing the first target data , and a class token that can represent the overall semantics of the first target data. The first processing model may be a model capable of performing feature extraction on the first target data, the model may be obtained by training the same type of data as the first target data, and the model may adopt a model structure capable of feature extraction in the related art, In the embodiments of the present application, the Transformer model is used as an example for description, but it is not limited to this. In the image-text search scenario, when the first target data is image data, the first processing model may be an image encoder; when the first target data is text data, the first processing model may be a text encoder.

In an optional embodiment, the first target data is first segmented to obtain multiple data blocks patchtoken, and then a learnable embedding vector class token is added to obtain an input sequence that is finally input to the first processing model, The class token may represent the overall semantics of the first target data. After the input sequence is input into the first processing model for feature extraction, a compressed vector, that is, the above-mentioned first feature vector can be obtained.

It should be noted that the first processing model may be deployed in the client or in the cloud server. In order to reduce the running resources of the client, in this embodiment of the present application, the first processing model is deployed in the cloud server as example to illustrate.

Step S506, using the second processing model to process the plurality of second target data respectively to obtain second feature vectors corresponding to the plurality of second target data.

Since the first target data and the second target data are two different types of data, it is necessary to train the first processing model and the second processing model respectively for the first target data and the second target data, so as to ensure the accuracy of feature extraction . In the image-text search scenario, when the second target data is text data, the second processing model may be a text encoder; when the second target data is image data, the second processing model may be an image encoder.

Since the processing purposes of the first processing model and the second processing model are the same, the structure of the second processing model is the same as that of the first processing model, and the processing flow of the second processing model is the same as the processing flow of the first processing model. I won't go into details.

Step S508, the first feature vector and the second feature vector are matched to determine the push data in the plurality of second target data, wherein the push data is used to represent the second target data matching the first target data.

In an optional embodiment, the similarity between the first feature vector and each second feature vector may be calculated separately, so as to match the first feature vector and each second feature vector, and further determine the corresponding minimum similarity The second target data is the second target data that matches the first target data. Therefore, the second target data can be determined as the push data that is finally pushed to the user.

The parameters of the first processing model and the second processing model are adjusted based on the global similarity between the first global feature and the second global feature, and the data block similarity between the first data block feature and the second data block feature. The feature and the first data block feature are obtained by processing the first training data by the first processing model, the second global feature and the second data block feature are obtained by processing the second training data by the second processing model, and the first training data and The second training data are different types of data, the first global feature is used to characterize the semantic feature of the first training data, the first data block feature is used to characterize the feature of the data block in the first training data, the first processing model and the The second processing model is a machine learning model.

It should be noted that, for the processing procedures of the first processing model and the second processing model, reference may be made to the model training method in 1 in the foregoing embodiment, and the specific training procedures will not be repeated here.

In the above embodiment of the present application, after the push data is determined, the method further includes: outputting the push data; receiving feedback data, wherein the feedback data is used to represent the data after the push data is modified; The parameters of the treatment model and the second treatment model are adjusted.

In an optional embodiment, when both the first processing model and the second processing model are deployed in the cloud server, after the push data is determined from the plurality of second target data, the push data can be fed back to the user , the user confirms, if the push data meets the user's needs, the user can feedback the confirmation information; if the push data does not meet the user's needs, the user can modify the push data to obtain the feedback data, and use the feedback data to modify the first processing model. and the parameters of the second processing model are adjusted to achieve the effect of improving server performance.

It should be noted that the preferred embodiments involved in the above embodiments of the present application are the same as the solutions, application scenarios, and implementation processes provided in Example 1, but are not limited to the solutions provided in Example 1.

Example 3

According to an embodiment of the present application, a data processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings may be executed in a computer system such as a set of computer-executable instructions, and although the flowchart A logical order is shown in the figures, but in some cases steps shown or described may be performed in an order different from that herein.

FIG. 6 is a flowchart of a second data processing method according to an embodiment of the present application. As shown in Figure 6, the method may include the following steps:

Step S602, displaying the first target data on the operation interface in response to the input instruction acting on the operation interface of the client.

The operation interface in the above steps can be an operation interface provided to the user in the client, and the user can provide the first target data by operating in the operation interface, and the first target data can be displayed on the first display of the operation interface. in the area. The input instruction can be an instruction for the user to input text content in the text box, or an instruction generated by the user by clicking the "shoot" button, or an instruction generated by the user by dragging the first target data to the input area, but not only limited to this.

Step S604, in response to the push instruction acting on the operation interface of the client, display the push data in the plurality of second target data on the operation interface, wherein the push data is used to represent the second target data that matches the first target data , the types of the first target data and the plurality of second target data are different, and the push data is obtained by matching the first feature vector corresponding to the first target data with the second feature vectors corresponding to the plurality of second target data. The vector is obtained by processing the first target data through the first processing model, and the second feature vector is obtained by processing the second target data through the second processing model.

The push instruction in the above steps may be an instruction generated by the user by clicking the "Push" button. In the image and text search scenario, the push instruction may be an image and text search instruction, but is not limited thereto.

The parameters of the first processing model and the second processing model are adjusted based on the global similarity between the first global feature and the second global feature, and the data block similarity between the first data block feature and the second data block feature. The feature and the first data block feature are obtained by processing the first training data by the first processing model, the second global feature and the second data block feature are obtained by processing the second training data by the second processing model, and the first training data and The second training data are different types of data, the first global feature is used to characterize the semantic feature of the first training data, the first data block feature is used to characterize the feature of the data block in the first training data, the first processing model and the The second processing model is a machine learning model.

It should be noted that the preferred embodiments involved in the above embodiments of the present application are the same as the solutions, application scenarios, and implementation processes provided in Example 2, but are not limited to the solutions provided in Example 2.

Example 4

According to the embodiments of the present application, a data processing method that can be applied to virtual reality scenarios such as virtual reality VR devices and augmented reality AR devices is also provided. It should be noted that the steps shown in the flowcharts of the accompanying drawings can be Execution in a computer system, such as a set of computer-executable instructions, and, although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that herein.

FIG. 7 is a flowchart of a third data processing method according to an embodiment of the present application. As shown in Figure 7, the method may include the following steps:

Step S702, displaying the first target data on the presentation screen of the virtual reality VR device or the augmented reality AR device.

Step S704: Acquire a plurality of second target data, wherein the types of the first target data and the plurality of second target data are different.

Step S706, using the first processing model to process the first target data to obtain a first feature vector corresponding to the first target data.

Step S708, using the second processing model to process the plurality of second target data respectively to obtain second feature vectors corresponding to the plurality of second target data.

Step S710: Match the first feature vector and the second feature vector to determine the push data in the plurality of second target data, wherein the push data is used to represent the second target data matching the first target data.

Step S712, driving the VR device or AR device to display the push data.

The parameters of the first processing model and the second processing model are adjusted based on the global similarity between the first global feature and the second global feature, and the data block similarity between the first data block feature and the second data block feature. The feature and the first data block feature are obtained by processing the first training data by the first processing model, the second global feature and the second data block feature are obtained by processing the second training data by the second processing model, and the first training data and The second training data are different types of data, the first global feature is used to characterize the semantic feature of the first training data, the first data block feature is used to characterize the feature of the data block in the first training data, the first processing model and the The second processing model is a machine learning model.

Optionally, in this embodiment, the above data processing method may be applied to a hardware environment composed of a server and a virtual reality device. To display the push data on the presentation screen of the virtual reality VR device or augmented reality AR device, the server may be the server corresponding to the media file operator. The above-mentioned networks include but are not limited to: wide area network, metropolitan area network or local area network. The above-mentioned virtual reality equipment does not Limited to: virtual reality helmet, virtual reality glasses, virtual reality all-in-one machine, etc.

Optionally, the virtual reality device includes: a memory, a processor and a transmission device. The memory is used for storing an application program, and the application program can be used for executing: displaying the first target data on the presentation screen of the virtual reality VR device or the augmented reality AR device; acquiring a plurality of second target data, wherein the first target data and The types of the plurality of second target data are different; the first target data is processed by the first processing model to obtain the first feature vector corresponding to the first target data; the plurality of second target data are processed by the second processing model respectively , obtain the second feature vectors corresponding to the plurality of second target data; match the first feature vector and the second feature vector to determine the push data in the plurality of second target data, wherein the push data is used to represent the The second target data that matches the target data; the VR device or the AR device is driven to display the push data; wherein, the parameters of the first processing model and the second processing model are based on the global similarity between the first global feature and the second global feature, the first The data block similarity between the data block feature and the second data block feature is adjusted. The first global feature and the first data block feature are obtained by processing the first training data through the first processing model, and the second global feature and the second data block are obtained. The feature is obtained by processing the second training data by the second processing model, the first training data and the second training data are different types of data, the first global feature is used to represent the semantic feature of the first training data, and the first data block feature For characterizing the features of the data blocks in the first training data, the first processing model and the second processing model are machine learning models.

It should be noted that the above data processing method applied in the VR device or AR device of this embodiment may include the method of the embodiment shown in FIG. 3 to achieve the purpose of driving the VR device or AR device to display push data.

Optionally, the processor in this embodiment may call the application program stored in the above-mentioned memory through the transmission device to execute the above-mentioned steps. The transmission device can receive the media files sent by the server through the network, and can also be used for data transmission between the above-mentioned processor and the memory.

Optionally, in a virtual reality device, a head-mounted display with eye-tracking, the screen in the HMD head-mounted display is used to display the displayed video picture, and the eye-tracking module in the HMD is used to obtain the real-time information of the user's eyeballs. Movement route, tracking system, used to track the user's position information and movement information in the real three-dimensional space, calculation processing unit, used to obtain the user's real-time position and movement information from the tracking system, and calculate the user's head in the virtual three-dimensional space. The three-dimensional coordinates in the virtual three-dimensional space, and the user's view orientation in the virtual three-dimensional space.

In the embodiment of the present application, the virtual reality device may be connected to the terminal, and the terminal and the server are connected through the network. The above virtual reality device is not limited to: virtual reality helmet, virtual reality glasses, virtual reality all-in-one machine, etc. Not limited to PCs, mobile phones, tablet computers, etc., the server may be a server corresponding to a media file operator, and the above-mentioned network includes but is not limited to: a wide area network, a metropolitan area network, or a local area network.

It should be noted that the preferred embodiments involved in the above embodiments of the present application are the same as the solutions, application scenarios, and implementation processes provided in Example 2, but are not limited to the solutions provided in Example 2.

Example 5

According to an embodiment of the present application, a model training method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings may be executed in a computer system such as a set of computer-executable instructions, and although the flowchart A logical order is shown in the figures, but in some cases steps shown or described may be performed in an order different from that herein.

FIG. 8 is a flowchart of another model training method according to an embodiment of the present application. As shown in Figure 8, the method may include the following steps:

Step S802, the server obtains the model training request sent by the client by calling the first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter is the model training request, and the model training request is used for processing the first model and the model training request. The second processing model is trained.

Step S804, the server obtains training samples based on the model training request, wherein the training samples include: different types of first training data and second training data.

Step S806, the server processes the first training data by using the first processing model to obtain a first global feature and a first data block feature, wherein the first global feature is used to represent the semantic feature of the first training data, and the first data block The features are used to characterize the features of the data blocks in the first training data.

Step S808, the server processes the second training data by using the second processing model to obtain the second global feature and the second data block feature.

Step S810, the server adjusts the parameters of the first processing model and the second processing model based on the global similarity between the first global feature and the second global feature, and the data block similarity between the first data block feature and the second data block feature. .

Step S812, the server outputs the first processing model and the second processing model to the client by calling the second interface, wherein the second interface includes a second parameter, and the parameter values of the second parameter are the first processing model and the second processing model, The first processing model and the second processing model are machine learning models.

It should be noted that the preferred embodiments involved in the above embodiments of the present application are the same as the solutions, application scenarios, and implementation processes provided in Example 1, but are not limited to the solutions provided in Example 1.

Example 6

According to an embodiment of the present application, a data processing method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings may be executed in a computer system such as a set of computer-executable instructions, and although the flowchart A logical order is shown in the figures, but in some cases steps shown or described may be performed in an order different from that herein.

FIG. 9 is a flowchart of a fourth data processing method according to an embodiment of the present application. As shown in Figure 9, the method may include the following steps:

Step S902, the server acquires the first target data sent by the client by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the first target data.

Step S904, the server acquires a plurality of second target data, wherein the types of the first target data and the plurality of second target data are different.

Step S906, the server processes the first target data by using the first processing model to obtain a first feature vector corresponding to the first target data.

Step S908, the server processes the plurality of second target data respectively by using the second processing model to obtain second feature vectors corresponding to the plurality of second target data.

Step S910, the server matches the first feature vector with the second feature vector, and determines the push data in the plurality of second target data, wherein the push data is used to represent the second target data matching the first target data.

Step S912, the server outputs the push data to the client by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the push data.

The parameters of the first processing model and the second processing model are adjusted based on the global similarity between the first global feature and the second global feature, and the data block similarity between the first data block feature and the second data block feature. The feature and the first data block feature are obtained by processing the first training data by the first processing model, the second global feature and the second data block feature are obtained by processing the second training data by the second processing model, and the first training data and The second training data are different types of data, the first global feature is used to characterize the semantic feature of the first training data, the first data block feature is used to characterize the feature of the data block in the first training data, the first processing model and the The second processing model is a machine learning model.

It should be noted that the preferred embodiments involved in the above embodiments of the present application are the same as the solutions, application scenarios, and implementation processes provided in Example 2, but are not limited to the solutions provided in Example 2.

Example 7

According to an embodiment of the present application, a model processing apparatus for implementing the above model training method is also provided. As shown in FIG. 10 , the apparatus 1000 includes: an acquisition module 1002 , a first processing module 1004 , a second processing module 1006 and Adjustment module 1008.

The acquisition module 1002 is used to acquire training samples, wherein the training samples include: different types of first training data and second training data; the first processing module 1004 is used to process the first training data by using the first processing model, obtaining the first global feature and the first data block feature, wherein the first global feature is used to represent the semantic feature of the first training data, and the first data block feature is used to represent the feature of the data block in the first training data; the second The processing module 1006 is used for using the second processing model to process the second training data to obtain the second global feature and the second data block feature; the adjusting module 1008 is used for the global similarity based on the first global feature and the second global feature, The second data block feature and the data block similarity of the second data block feature adjust the parameters of the first processing model and the second processing model, and the first processing model and the second processing model are machine learning models.

It should be noted here that the acquisition module 1002 , the first processing module 1004 , the second processing module 1006 and the adjustment module 1008 correspond to steps S302 to S308 in Embodiment 1, and the four modules and the corresponding steps are implemented by The examples and application scenarios are the same, but are not limited to the content disclosed in the above-mentioned Embodiment 1. It should be noted that, as a part of the apparatus, the above-mentioned modules may run in the computer terminal provided in Embodiment 1.

In the above embodiments of the present application, the adjustment module includes: a first construction unit, a second construction unit, a weighting unit, and an adjustment unit.

Wherein, the first construction unit is used to construct the first loss function based on the global similarity; the second construction unit is used to construct the second loss function based on the data block similarity; the weighting unit is used to obtain the difference between the first loss function and the second loss function The weighted sum is used to obtain the target loss function; the adjustment unit is used to adjust the parameters of the first processing model and the second processing model based on the target loss function.

In the above-mentioned embodiment of the present application, the first construction unit is further configured to obtain the first global similarity corresponding to the first sample and the second global similarity corresponding to the second sample from the global similarity, wherein the first same The two different types of data contained in this sample match, and the two different types of data contained in the second sample do not match; a first loss function is constructed based on the first global similarity and the second global similarity.

In the above-mentioned embodiment of the present application, the second construction unit is further configured to obtain the similarity of the first data block corresponding to the first sample and the similarity of the second data block corresponding to the second sample from the similarity of the data block; The similarity of a data block and the similarity of the second data block are used to construct a second loss function.

In the above embodiments of the present application, the apparatus further includes: a feature alignment module.

The feature alignment module is used to perform feature alignment on the second data block feature and the second data block feature by using the feature alignment module to obtain the data block similarity.

It should be noted that the preferred embodiments involved in the above embodiments of the present application are the same as the solutions, application scenarios, and implementation processes provided in Example 1, but are not limited to the solutions provided in Example 1.

Example 8

According to an embodiment of the present application, a data processing apparatus for implementing the above data processing method is also provided. As shown in FIG. 11 , the apparatus 1100 includes: an acquisition module 1102 , a first processing module 1104 , a second processing module 1106 and Matching module 1108.

The acquisition module 1102 is used to acquire different types of first target data and a plurality of second target data; the first processing module 1104 is used to process the first target data by using the first processing model to obtain the corresponding first target data. The first feature vector; the second processing module 1106 is used to process the plurality of second target data respectively by using the second processing model to obtain the second feature vectors corresponding to the plurality of second target data; the matching module 1108 is used to The feature vector and the second feature vector are matched to determine the push data in the plurality of second target data, wherein the push data is used to represent the second target data that matches the first target data, the first processing model and the second processing model The parameters of the model are adjusted based on the global similarity between the first global feature and the second global feature, the data block similarity between the first data block feature and the second data block feature, and the first global feature and the first data block feature are passed through the first The processing model is obtained by processing the first training data, the second global feature and the second data block feature are obtained by processing the second training data by the second processing model, and the first training data and the second training data are different types of data, The first global feature is used to represent the semantic feature of the first training data, the first data block feature is used to represent the feature of the data block in the first training data, and the first processing model and the second processing model are machine learning models.

It should be noted here that the acquisition module 1102 , the first processing module 1104 , the second processing module 1106 and the matching module 1108 correspond to steps S502 to S508 in Embodiment 2, and the four modules and the corresponding steps are implemented by The examples and application scenarios are the same, but are not limited to the content disclosed in the above-mentioned Embodiment 2. It should be noted that, as a part of the apparatus, the above-mentioned modules may run in the computer terminal provided in Embodiment 1.

Example 9

According to an embodiment of the present application, a data processing apparatus for implementing the above data processing method is also provided, which is deployed in a client. As shown in FIG. 12 , the apparatus 1200 includes: a first display module 1202 and a second display module 1204.

Among them, the first display module 1202 is used to display the first target data on the operation interface in response to the input command acting on the operation interface of the client; the second display module 1204 is used to respond to the push instruction acting on the operation interface, in the operation The interface displays the push data in the plurality of second target data, wherein the push data is used to represent the second target data that matches the first target data, and the first target data and the plurality of second target data are of different types. The data is obtained by matching the first feature vector corresponding to the first target data with the second feature vectors corresponding to a plurality of second target data, the first feature vector is obtained by processing the first target data by the first processing model, and the second feature vector is obtained by processing the first target data through the first processing model. The feature vector is obtained by processing the second target data through the second processing model, and the parameters of the first processing model and the second processing model are based on the global similarity of the first global feature and the second global feature, the first data block feature and the second The data block similarity of the data block feature is adjusted, the first global feature and the first data block feature are obtained by processing the first training data by the first processing model, and the second global feature and the second data block feature are obtained by the second processing model. The second training data is processed to obtain, the first training data and the second training data are different types of data, the first global feature is used to represent the semantic feature of the first training data, and the first data block feature is used to represent the first training data. The features of the data blocks in the data, the first processing model and the second processing model are machine learning models.

It should be noted here that the above-mentioned first display module 1202 and second display module 1204 correspond to steps S602 to S604 in Embodiment 3, and the two modules have the same instances and application scenarios as the corresponding steps, but they are different from each other. It is limited to the content disclosed in the above-mentioned Embodiment 3. It should be noted that, as a part of the apparatus, the above-mentioned modules may run in the computer terminal provided in Embodiment 1.

Example 10

According to an embodiment of the present application, a data processing apparatus for implementing the above data processing method is also provided. As shown in FIG. 13 , the apparatus 1300 includes: a first display module 1302, an acquisition module 1304, a first processing module 1306, The second processing module 1308 , the matching module 1310 and the second presentation module 1312 .

The first display module 1302 is used to display the first target data on the presentation screen of the virtual reality VR device or the augmented reality AR device; the acquisition module 1304 is used to obtain multiple second target data, wherein the first target data and the multiple The types of the second target data are different; the first processing module 1306 is used to process the first target data by using the first processing model to obtain the first feature vector corresponding to the first target data; the second processing module 1308 is used to use the first target data. The second processing model respectively processes the multiple second target data to obtain the second feature vectors corresponding to the multiple second target data; the matching module 1310 is configured to match the first feature vector and the second feature vector to determine multiple second feature vectors. The push data in the two target data, wherein the push data is used to represent the second target data that matches the first target data; the second display module 1312 is used to drive the VR device or AR device to display the push data; wherein, the first processing The parameters of the model and the second processing model are adjusted based on the global similarity between the first global feature and the second global feature, the data block similarity between the first data block feature and the second data block feature, and the first global feature and the first data The block feature is obtained by processing the first training data by the first processing model, and the second global feature and the second data block feature are obtained by processing the second training data by the second processing model. The first training data and the second training data are For different types of data, the first global feature is used to represent the semantic features of the first training data, the first data block feature is used to represent the features of the data blocks in the first training data, and the first processing model and the second processing model are machines. Learning models.

It should be noted here that the above-mentioned first display module 1302 , acquisition module 1304 , first processing module 1306 , second processing module 1308 , matching module 1310 , and second display module 1312 correspond to steps S702 to S702 in Embodiment 4 S712, the six modules and the corresponding steps have the same examples and application scenarios, but are not limited to the content disclosed in the foregoing Embodiment 4. It should be noted that, as a part of the apparatus, the above-mentioned modules may run in the computer terminal provided in Embodiment 1.

Example 11

According to an embodiment of the present application, a model training device for implementing the above model training method is also provided, which is deployed on a server. As shown in FIG. 14 , the device 1400 includes: a first calling module 1402 , an obtaining module 1404 , a first A processing module 1406 , a second processing module 1408 , an adjustment module 1410 and a second calling module 1412 .

The first calling module 1402 is used to obtain the model training request sent by the client by calling the first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter is the model training request, and the model training request is used to The first processing model and the second processing model are trained; the obtaining module 1404 is used to obtain training samples based on the model training request, wherein the training samples include: different types of first training data and second training data; the first processing module 1406 uses is used to process the first training data by using the first processing model to obtain a first global feature and a first data block feature, wherein the first global feature is used to represent the semantic feature of the first training data, and the first data block feature is used for Characterize the features of the data blocks in the first training data; the second processing module 1408 is used to process the second training data by using the second processing model to obtain second global features and second data block features; the adjustment module 1410 is used to The global similarity of the first global feature and the second global feature, the data block similarity of the first data block feature and the second data block feature, adjust the parameters of the first processing model and the second processing model; the second calling module 1412 is used to output the first processing model and the second processing model to the client by calling the second interface, wherein the second interface includes a second parameter, and the parameter values of the second parameter are the first processing model and the second processing model; wherein , the parameters of the first processing model and the second processing model are adjusted based on the global similarity of the first global feature and the second global feature, the data block similarity of the first data block feature and the second data block feature, and the first global feature and the first data block feature are obtained by processing the first training data by the first processing model, the second global feature and the second data block feature are obtained by processing the second training data by the second processing model, the first training data and the second data block feature are obtained by processing the second training data. The second training data are different types of data, the first global feature is used to characterize the semantic features of the first training data, the first data block feature is used to characterize the features of the data blocks in the first training data, the first processing model and the second The processing model is a machine learning model.

It should be noted here that the above-mentioned first calling module 1402 , obtaining module 1404 , first processing module 1406 , second processing module 1408 , adjustment module 1410 and second calling module 1412 correspond to steps S802 to S802 in Embodiment 5 S812, the six modules and the corresponding steps have the same examples and application scenarios, but are not limited to the content disclosed in Embodiment 5 above. It should be noted that, as a part of the apparatus, the above-mentioned modules may run in the computer terminal provided in Embodiment 1.

Example 12

According to an embodiment of the present application, a data processing apparatus for implementing the above data processing method is also provided, which is deployed on a server. As shown in FIG. 15 , the apparatus 1500 includes: a first calling module 1502 , an obtaining module 1504 , a first A processing module 1506 , a second processing module 1508 , a matching module 1510 and a second calling module 1512 .

The first calling module 1502 is configured to obtain the first target data sent by the client by calling the first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the first target data; the obtaining module 1504 uses for obtaining a plurality of second target data, wherein the types of the first target data and the plurality of second target data are different; the first processing module 1506 is used to process the first target data by using the first processing model to obtain the first target data The first feature vector corresponding to the data; the second processing module 1508 is used to process the plurality of second target data respectively by using the second processing model to obtain the second feature vector corresponding to the plurality of second target data; the matching module 1510 is used for Matching the first feature vector and the second feature vector to determine the push data in the plurality of second target data, wherein the push data is used to represent the second target data that matches the first target data; the second calling module 1512 It is used to output the push data to the client by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the push data.

The parameters of the first processing model and the second processing model are adjusted based on the global similarity between the first global feature and the second global feature, and the data block similarity between the first data block feature and the second data block feature. The feature and the first data block feature are obtained by processing the first training data by the first processing model, the second global feature and the second data block feature are obtained by processing the second training data by the second processing model, and the first training data and The second training data are different types of data, the first global feature is used to characterize the semantic feature of the first training data, the first data block feature is used to characterize the feature of the data block in the first training data, the first processing model and the The second processing model is a machine learning model.

It should be noted here that the above-mentioned first calling module 1502 , obtaining module 1504 , first processing module 1506 , second processing module 1508 , matching module 1510 and second calling module 1512 correspond to steps S902 to S902 in Embodiment 6 S912, the six modules and the corresponding steps have the same examples and application scenarios, but are not limited to the content disclosed in the above-mentioned Embodiment 6. It should be noted that, as a part of the apparatus, the above-mentioned modules may run in the computer terminal provided in Embodiment 1.

Example 13

Embodiments of the present application may provide a computer terminal, and the computer terminal may be any computer terminal in a computer terminal cluster. Optionally, in this embodiment, the above-mentioned computer terminal may also be replaced by a terminal device such as a mobile terminal.

Optionally, in this embodiment, the above-mentioned computer terminal may be located in at least one network device among multiple network devices of a computer network.

In this embodiment, the above-mentioned computer terminal can execute the program code of the following steps in the model training method: obtain training samples, the training samples include: different types of first training data and second training data; The training data is processed to obtain a first global feature and a first data block feature. The first global feature is used to represent the semantic feature of the first training data, and the first data block feature is used to represent the feature of the data block in the first training data. Utilize the second processing model to process the second training data to obtain the second global feature and the second data block feature; Based on the global similarity of the first global feature and the second global feature, the first data block feature and the second data The data block similarity of the block feature adjusts the parameters of the first processing model and the second processing model, and the first processing model and the second processing model are machine learning models.

Optionally, FIG. 16 is a structural block diagram of a computer terminal according to an embodiment of the present application. As shown in FIG. 16 , the computer terminal A may include: one or more (only one is shown in the figure) processors 1602 and a memory 1604 .

The memory can be used to store software programs and modules, such as the model training method and device in the embodiments of the present application, and program instructions/modules corresponding to the data processing method and device, and the processor runs the software programs and modules stored in the memory by running the software programs and modules. , so as to perform various functional applications and data processing, that is, to implement the above-mentioned model training method and data processing method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory located remotely from the processor, and these remote memories may be connected to Terminal A through a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: acquire training samples, the training samples include: different types of first training data and second training data; The training data is processed to obtain a first global feature and a first data block feature. The first global feature is used to represent the semantic feature of the first training data, and the first data block feature is used to represent the feature of the data block in the first training data. Utilize the second processing model to process the second training data to obtain the second global feature and the second data block feature; Based on the global similarity of the first global feature and the second global feature, the first data block feature and the second data The data block similarity of the block feature adjusts the parameters of the first processing model and the second processing model, and the first processing model and the second processing model are machine learning models.

Optionally, the above-mentioned processor may also execute the program code of the following steps: constructing a first loss function based on the global similarity; constructing a second loss function based on the data block similarity; obtaining a weighted sum of the first loss function and the second loss function , the target loss function is obtained; the parameters of the first processing model and the second processing model are adjusted based on the target loss function.

Optionally, the above-mentioned processor may also execute the program code of the following steps: obtaining the first global similarity corresponding to the first sample and the second global similarity corresponding to the second sample from the global similarity, wherein the first Two different types of data contained in the sample match, and two different types of data contained in the second sample do not match; a first loss function is constructed based on the first global similarity and the second global similarity.

Optionally, the above-mentioned processor may also execute the program code of the following steps: obtaining the similarity of the first data block corresponding to the first sample and the similarity of the second data block corresponding to the second sample from the similarity of the data block; The similarity of the first data block and the similarity of the second data block are used to construct a second loss function.

Optionally, the above-mentioned processor may further execute the program code of the following steps: using a feature alignment module to perform feature alignment on the features of the first data block and the features of the second data block to obtain the similarity of the data blocks.

The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: acquiring different types of first target data and multiple second target data; using the first processing model to process the first target data, Obtain the first feature vector corresponding to the first target data; use the second processing model to process the plurality of second target data respectively to obtain the second feature vector corresponding to the plurality of second target data; The feature vector is matched to determine the push data in the plurality of second target data, wherein the push data is used to represent the second target data that matches the first target data; wherein, the parameters of the first processing model and the second processing model The adjustment is made based on the global similarity of the first global feature and the second global feature, and the data block similarity of the first data block feature and the second data block feature, and the first global feature and the first data block feature are adjusted by the first processing model. The first training data is processed and obtained, the second global feature and the second data block feature are obtained by processing the second training data through the second processing model, the first training data and the second training data are different types of data, the first global feature The feature is used to represent the semantic feature of the first training data, the first data block feature is used to represent the feature of the data block in the first training data, and the first processing model and the second processing model are machine learning models.

The processor can call the information and the application program stored in the memory through the transmission device to perform the following steps: display the first target data on the operation interface in response to the input instruction acting on the operation interface of the client; A push instruction, displaying the push data in the plurality of second target data on the operation interface, wherein the push data is used to represent the second target data matching the first target data, the first target data and the plurality of second target data The push data is obtained by matching the first feature vector corresponding to the first target data with the second feature vectors corresponding to multiple second target data, and the first feature vector is processed by the first processing model on the first target data. The second feature vector is obtained by processing the second target data through the second processing model, and the parameters of the first processing model and the second processing model are based on the global similarity of the first global feature and the second global feature, the first data The data block similarity between the block feature and the second data block feature is adjusted. The first global feature and the first data block feature are obtained by processing the first training data through the first processing model, and the second global feature and the second data block feature are obtained. The second training data is processed by the second processing model. The first training data and the second training data are different types of data. The first global feature is used to represent the semantic feature of the first training data, and the first data block feature is used for For characterizing the features of the data blocks in the first training data, the first processing model and the second processing model are machine learning models.

The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: displaying the first target data on the presentation screen of the virtual reality VR device or the augmented reality AR device; acquiring a plurality of second target data, wherein , the types of the first target data and the plurality of second target data are different; use the first processing model to process the first target data to obtain the first feature vector corresponding to the first target data; use the second processing model to process the multiple The second target data is processed to obtain a plurality of second feature vectors corresponding to the second target data; the first feature vector and the second feature vector are matched to determine the push data in the plurality of second target data, wherein the push data Used to characterize the second target data that matches the first target data; drive the VR device or AR device to display the push data; wherein, the parameters of the first processing model and the second processing model are based on the first global feature and the second global feature. The global similarity, the data block similarity of the first data block feature and the second data block feature are adjusted. The first global feature and the first data block feature are obtained by processing the first training data by the first processing model. The feature and the second data block feature are obtained by processing the second training data by the second processing model. The first training data and the second training data are different types of data, and the first global feature is used to represent the semantic feature of the first training data. , the first data block feature is used to represent the feature of the data block in the first training data, and the first processing model and the second processing model are machine learning models.

The processor can call the information and the application program stored in the memory through the transmission device to perform the following steps: obtain the model training request sent by the client by calling the first interface, wherein the first interface includes a first parameter, the parameter of the first parameter The value is a model training request, and the model training request is used to train the first processing model and the second processing model; training samples are obtained based on the model training request, wherein the training samples include: different types of first training data and second training data Utilize the first processing model to process the first training data to obtain the first global feature and the first data block feature, wherein the first global feature is used to characterize the semantic feature of the first training data, and the first data block feature is used for Characterize the feature of the data block in the first training data; use the second processing model to process the second training data to obtain the second global feature and the second data block feature; based on the global similarity of the first global feature and the second global feature adjust the parameters of the first processing model and the second processing model; output the first processing model and the second processing model by calling the second interface to The client, wherein the second interface includes a second parameter, and the parameter values of the second parameter are the first processing model and the second processing model; wherein the parameters of the first processing model and the second processing model are based on the first global feature and the first processing model. The global similarity of the second global feature, the data block similarity of the first data block feature and the second data block feature are adjusted, and the first global feature and the first data block feature are obtained by processing the first training data through the first processing model. , the second global feature and the second data block feature are obtained by processing the second training data by the second processing model, the first training data and the second training data are different types of data, and the first global feature is used to represent the first training data Semantic features of the data, the first data block feature is used to represent the feature of the data block in the first training data, and the first processing model and the second processing model are machine learning models.

The processor can call the information and the application program stored in the memory through the transmission device to perform the following steps: obtain the first target data sent by the client by calling the first interface, wherein the first interface includes a first parameter, and the first parameter The parameter value is the first target data; a plurality of second target data are obtained, wherein the types of the first target data and the plurality of second target data are different; the first target data is processed by using the first processing model to obtain the first target data The first feature vector corresponding to the data; the second processing model is used to process the multiple second target data respectively to obtain the second feature vector corresponding to the multiple second target data; the first feature vector and the second feature vector are matched , determine the push data in the plurality of second target data, wherein the push data is used to represent the second target data that matches the first target data; output the push data to the client by calling the second interface, wherein the second interface Including a second parameter, the parameter value of the second parameter is the push data; wherein, the parameters of the first processing model and the second processing model are based on the global similarity of the first global feature and the second global feature, the first data block feature and the first data block feature. The data block similarity of the two data block features is adjusted, the first global feature and the first data block feature are obtained by processing the first training data through the first processing model, and the second global feature and the second data block feature are obtained through the second processing. The model processes the second training data to obtain, the first training data and the second training data are different types of data, the first global feature is used to represent the semantic feature of the first training data, and the first data block feature is used to represent the first The features of the data blocks in the training data, the first processing model and the second processing model are machine learning models.

Using the embodiments of the present application, a model training solution is provided. Use the first processing model to process the first training data to obtain first global features and first data block features, and use the second processing model to process the second training data to obtain second global features and second data block features Finally, based on the global similarity of the first global feature and the second global feature, the data block similarity of the first data block feature and the second data block feature, the parameters of the first processing model and the second processing model are adjusted to achieve The purpose of training the feature extraction model is to achieve a more accurate expression of the similarity between images and texts, improve the training effect of the model, and further improve the technical effect of the accuracy of the model's processing of multi-modal data, thereby solving the related problems. The technical problem of poor accuracy in processing multimodal data through models in technology. In the image and text search scenario, the first processing model and the second processing model can extract image features and text features with higher accuracy, further making the searched information more accurate and more in line with the user's search needs, thus achieving The effect of improving the accuracy of image and text search.

Those of ordinary skill in the art can understand that the structure shown in FIG. 16 is for illustration only, and the computer terminal can also be a smart phone (such as an Android mobile phone, an iOS mobile phone, etc.), a tablet computer, an applause computer, and a mobile Internet device (Mobile Internet Devices, MID), Terminal equipment such as PAD. FIG. 16 does not limit the structure of the above electronic device. For example, the computer terminal A may also include more or less components than those shown in FIG. 16 (eg, a network interface, a display device, etc.), or have a different configuration than that shown in FIG. 16 .

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium can Including: flash disk, read-only memory (Read-Only Memory, ROM), random access device (RandomAccess Memory, RAM), magnetic disk or optical disk, etc.

Example 14

Embodiments of the present application also provide a computer-readable storage medium. Optionally, in this embodiment, the above-mentioned computer-readable storage medium may be used to store program codes executed by the model training method and the data processing method provided in the above-mentioned embodiment.

Optionally, in this embodiment, the computer-readable storage medium may be located in any computer terminal in a computer terminal cluster in a computer network, or in any mobile terminal in a mobile terminal group.

Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for performing the following steps: acquiring training samples, the training samples including: different types of first training data and second training data; using The first processing model processes the first training data to obtain a first global feature and a first data block feature. The first global feature is used to represent the semantic feature of the first training data, and the first data block feature is used to represent the first training data. Features of the data blocks in the data; using the second processing model to process the second training data to obtain the second global feature and the second data block feature; based on the global similarity of the first global feature and the second global feature, the first The data block similarity between the data block feature and the second data block feature adjusts the parameters of the first processing model and the second processing model, and the first processing model and the second processing model are machine learning models.

Optionally, the above-mentioned computer-readable storage medium is further configured to store program codes for performing the following steps: constructing a first loss function based on global similarity; constructing a second loss function based on data block similarity; obtaining the first loss function The weighted sum of the second loss function and the target loss function is obtained; the parameters are adjusted based on the target loss function.

Optionally, the above-mentioned computer-readable storage medium is further configured to store program codes for performing the following steps: obtaining a first global similarity corresponding to the first sample from the global similarity, and a second corresponding to the second sample from the global similarity. The global similarity, wherein the two different types of data contained in the first sample match, and the two different types of data contained in the second sample do not match; based on the first global similarity and the second global similarity, construct the first a loss function.

Optionally, the above-mentioned computer-readable storage medium is further configured to store program codes for performing the following steps: obtaining the similarity of the first data block corresponding to the first sample from the similarity of the data block, and the corresponding degree of the second sample. The second data block similarity; based on the first data block similarity and the second data block similarity, a second loss function is constructed.

Optionally, the computer-readable storage medium is further configured to store program codes for performing the following steps: using a feature alignment module to perform feature alignment on the first data block feature and the second data block feature to obtain the data block similarity.

The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: acquiring different types of first target data and multiple second target data; using the first processing model to process the first target data, Obtain the first feature vector corresponding to the first target data; use the second processing model to process the plurality of second target data respectively to obtain the second feature vector corresponding to the plurality of second target data; The feature vector is matched to determine the push data in the plurality of second target data, wherein the push data is used to represent the second target data that matches the first target data; wherein, the parameters of the first processing model and the second processing model The adjustment is made based on the global similarity of the first global feature and the second global feature, and the data block similarity of the first data block feature and the second data block feature, and the first global feature and the first data block feature are adjusted by the first processing model. The first training data is processed and obtained, the second global feature and the second data block feature are obtained by processing the second training data through the second processing model, the first training data and the second training data are different types of data, the first global feature The feature is used to represent the semantic feature of the first training data, the first data block feature is used to represent the feature of the data block in the first training data, and the first processing model and the second processing model are machine learning models.

Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: displaying the first target data on the operation interface in response to an input instruction acting on the operation interface of the client; In response to the push instruction acting on the operation interface, the push data in the plurality of second target data is displayed on the operation interface, wherein the push data is used to represent the second target data matching the first target data, and the first target data Different from the types of multiple second target data, the push data is obtained by matching the first feature vector corresponding to the first target data with the second feature vectors corresponding to the multiple second target data, and the first feature vector is processed by the first process. The model is obtained by processing the first target data, the second feature vector is obtained by processing the second target data by the second processing model, and the parameters of the first processing model and the second processing model are based on the first global feature and the second global feature. The global similarity, the data block similarity of the first data block feature and the second data block feature are adjusted. The first global feature and the first data block feature are obtained by processing the first training data by the first processing model. The feature and the second data block feature are obtained by processing the second training data by the second processing model. The first training data and the second training data are different types of data, and the first global feature is used to represent the semantic feature of the first training data. , the first data block feature is used to represent the feature of the data block in the first training data, and the first processing model and the second processing model are machine learning models.

Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for performing the following steps: displaying the first target data on a presentation screen of a virtual reality VR device or an augmented reality AR device; acquiring multiple a second target data, wherein the types of the first target data and the plurality of second target data are different; use the first processing model to process the first target data to obtain the first feature vector corresponding to the first target data; use the first target data to process the first target data; The second processing model respectively processes multiple second target data to obtain second feature vectors corresponding to the multiple second target data; matches the first feature vector and the second feature vector to determine which of the multiple second target data push data, wherein the push data is used to represent second target data that matches the first target data; drive the VR device or AR device to display the push data; wherein the parameters of the first processing model and the second processing model are based on the first global The global similarity of the feature and the second global feature, the data block similarity of the first data block feature and the second data block feature are adjusted, and the first global feature and the first data block feature are processed by the first processing model. After processing, the second global feature and the second data block feature are obtained by processing the second training data through the second processing model. The first training data and the second training data are different types of data, and the first global feature is used to represent The semantic feature of the first training data, the first data block feature is used to represent the feature of the data block in the first training data, and the first processing model and the second processing model are machine learning models.

Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: obtaining a model training request sent by the client by invoking a first interface, wherein the first interface includes a first interface. parameter, the parameter value of the first parameter is a model training request, and the model training request is used to train the first processing model and the second processing model; training samples are obtained based on the model training request, wherein the training samples include: different types of first processing models. training data and second training data; using the first processing model to process the first training data to obtain a first global feature and a first data block feature, wherein the first global feature is used to represent the semantic feature of the first training data, The first data block feature is used to characterize the feature of the data block in the first training data; the second training data is processed by the second processing model to obtain the second global feature and the second data block feature; based on the first global feature and The global similarity of the second global feature, the data block similarity of the first data block feature and the second data block feature, adjust the parameters of the first processing model and the second processing model; output the first processing by calling the second interface The model and the second processing model are sent to the client, wherein the second interface includes a second parameter, the parameter values of the second parameter are the first processing model and the second processing model, and the first processing model and the second processing model are machine learning models .

Optionally, in this embodiment, the computer-readable storage medium is configured to store program codes for executing the following steps: acquiring the first target data sent by the client by invoking a first interface, wherein the first interface includes a first interface. a parameter, the parameter value of the first parameter is the first target data; obtain a plurality of second target data, wherein the types of the first target data and the plurality of second target data are different; use the first processing model to analyze the first target data processing to obtain a first feature vector corresponding to the first target data; using the second processing model to process a plurality of second target data respectively to obtain a second feature vector corresponding to the plurality of second target data; Matching with the second feature vector to determine the push data in the plurality of second target data, wherein the push data is used to represent the second target data that matches the first target data; output the push data by calling the second interface, wherein , the second interface includes a second parameter, and the parameter value of the second parameter is the push data to the client; wherein, the parameters of the first processing model and the second processing model are based on the global similarity of the first global feature and the second global feature, the first The data block similarity between a data block feature and a second data block feature is adjusted. The first global feature and the first data block feature are obtained by processing the first training data through the first processing model. The second global feature and the second data The block feature is obtained by processing the second training data by the second processing model, the first training data and the second training data are different types of data, the first global feature is used to represent the semantic feature of the first training data, and the first data block The features are used to characterize the features of the data blocks in the first training data, and the first processing model and the second processing model are machine learning models.

The above-mentioned serial numbers of the embodiments of the present application are only for description, and do not represent the advantages or disadvantages of the embodiments.

In the above-mentioned embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The apparatus embodiments described above are only illustrative, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components may be combined or Integration into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some interfaces, indirect coupling or communication connection of units or modules, and may be in electrical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware, or may be implemented in the form of software functional units.

The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, and the computer software products are stored in a storage medium , including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes .

The above are only the preferred embodiments of the present application. It should be pointed out that for those skilled in the art, without departing from the principles of the present application, several improvements and modifications can also be made. It should be regarded as the protection scope of this application.

Claims (12)

1. A method of model training, comprising:

obtaining training samples, wherein the training samples comprise: different types of first training data and second training data;

processing the first training data by using a first processing model to obtain a first global feature and a first data block feature, wherein the first global feature is used for representing semantic features of the first training data, and the first data block feature is used for representing features of data blocks in the first training data;

processing the second training data by using a second processing model to obtain a second global feature and a second data block feature;

adjusting parameters of the first processing model and the second processing model based on the global similarity of the first global feature and the second global feature and the data block similarity of the first data block feature and the second data block feature, wherein the first processing model and the second processing model are machine learning models.

2. The method of claim 1, wherein adjusting parameters of the first and second processing models based on the global similarity of the first and second global features and the block similarity of the first and second block features comprises:

constructing a first loss function based on the global similarity;

constructing a second loss function based on the similarity of the data blocks;

obtaining a weighted sum of the first loss function and the second loss function to obtain a target loss function;

adjusting parameters of the first and second process models based on the objective loss function.

3. The method of claim 2, wherein constructing a first loss function based on the global similarity comprises:

acquiring a first global similarity corresponding to a first sample and a second global similarity corresponding to a second sample from the global similarities, wherein the two different types of data contained in the first sample are matched, and the two different types of data contained in the second sample are not matched;

constructing the first loss function based on the first global similarity and the second global similarity.

4. The method of claim 2, wherein constructing a second loss function based on the data block similarity comprises:

acquiring a first data block similarity corresponding to a first sample and a second data block similarity corresponding to a second sample from the data block similarities;

and constructing the second loss function based on the first data block similarity and the second data block similarity.

5. The method of claim 1, further comprising:

and performing feature alignment on the first data block features and the second data block features by using a feature alignment module to obtain the data block similarity.

6. The method according to any one of claims 1 to 5, wherein the first training data is image data and the second training data is text data.

7. A data processing method, comprising:

acquiring different types of first target data and a plurality of second target data;

processing the first target data by using a first processing model to obtain a first feature vector corresponding to the first target data;

respectively processing the plurality of second target data by using a second processing model to obtain second feature vectors corresponding to the plurality of second target data;

matching the first feature vector and the second feature vector, and determining push data in the plurality of second target data, wherein the push data is used for representing the second target data matched with the first target data;

the parameters of the first processing model and the second processing model are adjusted based on the global similarity of a first global feature and a second global feature, and the data block similarity of the first data block feature and a second data block feature, the first global feature and the first data block feature are obtained by processing first training data through the first processing model, the second global feature and the second data block feature are obtained by processing second training data through the second processing model, the first training data and the second training data are different types of data, the first global feature is used for representing the semantic feature of the first training data, the first data block feature is used for representing the feature of a data block in the first training data, and the first processing model and the second processing model are machine learning models.

8. A data processing method, comprising:

responding to an input instruction acted on an operation interface of a client, and displaying first target data on the operation interface;

displaying, on the operation interface, push data in a plurality of second target data in response to a push instruction acting on the operation interface, wherein the push data is used for characterizing the second target data matching with the first target data, the types of the first target data and the plurality of second target data are different, the push data is obtained by matching a first feature vector corresponding to the first target data with a second feature vector corresponding to the plurality of second target data, the first feature vector is obtained by processing the first target data through a first processing model, the second feature vector is obtained by processing the second target data through a second processing model, parameters of the first processing model and the second processing model are adjusted based on global similarities of first global features and second global features, data block similarities of first data block features and second data block features, the first global features and the first data block features are obtained by processing the first processing model on different types of first data, the second feature block features and the second data block features are obtained by processing the training data through the first processing model, the training data of the first global features and the training data of the second data block features are obtained by processing the training data blocks, and the training data is used for characterizing the training data of the first target data and the training data.

9. A method of model training, comprising:

the method comprises the steps that a server obtains a model training request sent by a client by calling a first interface, wherein the first interface comprises a first parameter, the parameter value of the first parameter is the model training request, and the model training request is used for training a first processing model and a second processing model;

the server obtains training samples based on the model training request, wherein the training samples comprise: different types of first training data and second training data;

the server processes the first training data by using the first processing model to obtain a first global feature and a first data block feature, wherein the first global feature is used for representing a semantic feature of the first training data, and the first data block feature is used for representing a feature of a data block in the first training data;

the server processes the second training data by using the second processing model to obtain a second global feature and a second data block feature;

the server adjusts parameters of the first processing model and the second processing model based on the global similarity of the first global feature and the second global feature and the data block similarity of the first data block feature and the second data block feature;

the server outputs the first processing model and the second processing model to the client by calling a second interface, wherein the second interface comprises a second parameter, a parameter value of the second parameter is the first processing model and the second processing model, and the first processing model and the second processing model are machine learning models.

10. A data processing method, comprising:

the method comprises the steps that a server obtains first target data sent by a client by calling a first interface, wherein the first interface comprises a first parameter, and a parameter value of the first parameter is the first target data;

the server acquires a plurality of second target data, wherein the types of the first target data and the second target data are different;

the server processes the first target data by using a first processing model to obtain a first feature vector corresponding to the first target data;

the server respectively processes the second target data by using a second processing model to obtain second feature vectors corresponding to the second target data;

the server matches the first feature vector with the second feature vector, and determines push data in the second target data, wherein the push data is used for representing the second target data matched with the first target data;

the server outputs the push data to the client by calling a second interface, wherein the second interface comprises a second parameter, and a parameter value of the second parameter is the push data;

the parameters of the first processing model and the second processing model are adjusted based on the global similarity of a first global feature and a second global feature, and the data block similarity of the first data block feature and a second data block feature, the first global feature and the first data block feature are obtained by processing first training data through the first processing model, the second global feature and the second data block feature are obtained by processing second training data through the second processing model, the first training data and the second training data are different types of data, the first global feature is used for representing the semantic feature of the first training data, the first data block feature is used for representing the feature of a data block in the first training data, and the first processing model and the second processing model are machine learning models.

11. A computer-readable storage medium, comprising a stored program, wherein the program, when executed, controls an apparatus in which the computer-readable storage medium is located to perform the method of any one of claims 1 to 10.

12. A computer terminal, comprising:

a processor;

a memory coupled to the processor for providing instructions to the processor for processing the following processing steps: obtaining training samples, wherein the training samples comprise: different types of first training data and second training data; processing the first training data by using a first processing model to obtain a first global feature and a first data block feature, wherein the first global feature is used for representing semantic features of the first training data, and the first data block feature is used for representing features of data blocks in the first training data; processing the second training data by using a second processing model to obtain a second global feature and a second data block feature; adjusting parameters of the first processing model and the second processing model based on the global similarity of the first global feature and the second global feature and the data block similarity of the first data block feature and the second data block feature, wherein the first processing model and the second processing model are machine learning models.

Images