CN118468138A - A multimodal sentiment analysis model construction method, analysis model and analysis method - Google Patents

Abstract

The invention discloses a multi-mode emotion analysis model construction method, an analysis model and an analysis method, wherein the construction method comprises the following steps: obtaining a pre-training language sub-model and a training set; extracting sample features in a training set, constructing a multi-modal expert in a pre-training language sub-model, and introducing the multi-modal features into the pre-training language sub-model to obtain the output of the multi-modal expert; adding the mixed output to an output of a transducer layer of the pre-trained language sub-model; predicting emotion by adopting a preset MLP classifier to obtain a target model; and taking the trained target model as an emotion analysis model. By inserting the multi-modal expert into the pre-training language sub-model, the problem of overlarge parameter caused by using the pre-training language sub-model in multi-modal emotion analysis is solved; the characteristics of different modes at different times are analyzed through the multi-mode routing, and the token is distributed to an appropriate expert for processing, so that the judgment on the validity of the multi-mode characteristics is increased, and the classification accuracy is improved.

Description

A multimodal sentiment analysis model construction method, analysis model and analysis method

Technical Field

The present application relates to the field of model training technology, and specifically to a multimodal sentiment analysis model construction method, an analysis model, and an analysis method.

Background Art

The goal of sentiment analysis is to identify emotional expressions from human conversations. With the development of pre-trained language sub-models and related Transformer structures, sentiment analysis of text has made significant progress. However, processing text information alone cannot accurately express human emotions, because it contains not only text descriptions, but also body movements, expressions, and tone of voice. These different modal information can jointly reflect the speaker's emotions. To solve this problem, the task of multimodal sentiment analysis (MSA) was proposed. It aims to integrate multimodal features to analyze emotions.

First, the method of fully fine-tuning pre-trained models is often used in multimodal sentiment analysis (MSA) tasks. Although this method has great advantages over training from scratch, as the amount of data and the number of pre-trained model parameters increase, using this method to train the model will bring too many parameters that need to be trained, resulting in excessive resource consumption. Secondly, existing fusion methods often only focus on how to achieve a more ideal fusion. However, multimodal sentiment analysis requires analyzing the emotions hidden in video and audio, both of which often contain more noise and misleading content, such as the movement of people in the background of the video and environmental noise in the audio. Analyzing these irrelevant contents will have an adverse effect on model prediction.

A common solution in natural language processing (NLP) is parameter efficient fine tuning (PEFT), which can achieve the migration of pre-trained models to various downstream tasks by introducing a small number of new parameters into the pre-trained language sub-model. However, these PEFT methods only support unimodal data, which limits their application in multimodal sentiment analysis. In addition, existing multimodal sentiment analysis methods often align features before fusion. However, they ignore the analysis of different temporal features of different modalities and do not filter invalid features before fusion.

Based on the above analysis, how to reduce the number of parameters that need to be trained in the model and judge the validity of multimodal information before feature fusion to improve the training efficiency and accuracy of the model is an urgent problem to be solved in multimodal sentiment analysis.

Summary of the invention

In this embodiment, a multimodal sentiment analysis model construction method, an analysis model and an analysis method are provided to solve the problem that the application of multimodal sentiment analysis in the related art is limited and invalid features are not filtered.

In a first aspect, the present invention provides a method for constructing a multimodal sentiment analysis model, the method comprising:

Obtaining a pre-trained language sub-model and a training set, wherein the training set includes a plurality of labeled training samples;

Extracting features of samples in the training set to obtain multimodal features and aligning the multimodal features;

Constructing a multimodal expert in a pre-trained language sub-model, introducing the multimodal features into the pre-trained language sub-model, and obtaining an output of the multimodal expert;

Calculating the gating value of the multimodal expert, integrating the output of the multimodal expert through discrete routing, taking the weighted sum of the output of the multimodal expert and the gating value as the output of the hybrid expert network, and adding the hybrid output to the output of the Transformer layer of the pre-trained language sub-model;

Use the preset MLP classifier to predict sentiment and obtain the target model;

The training samples are input into the target model for classification training, and the trained target model is used as a sentiment analysis model.

In an optional embodiment, extracting sample features from the training set to obtain multimodal features and aligning the multimodal features includes:

Acquire text features of samples in a training set through the pre-trained language sub-model;

The samples in the training set are encoded by a pre-trained encoder to obtain underlying features, where the underlying features include video features and audio features.

In an optional embodiment, a multimodal expert is constructed in a pre-trained language sub-model, and the multimodal features are introduced into the pre-trained language sub-model to obtain the output of the multimodal expert, including:

The multimodal expert input at the i-th word is recorded as a triple, and the multimodal expert input is obtained by calculation;

Capturing the representation of the fusion process through multimodal experts;

The fused representation is mapped to the embedding dimension of the pre-trained language sub-model through the up-projection matrix to obtain the output of the multimodal expert.

In an optional embodiment, calculating a gating value of a multimodal expert, integrating the output of the multimodal expert, taking a weighted sum of the output of the multimodal expert and the gating value as the output of a hybrid expert network, and adding the hybrid output to the output of a Transformer layer of a pre-trained language sub-model includes:

Calculate the gating value of the multimodal expert through the multimodal gate;

Integrate the output of multimodal experts through discrete routing;

Taking the weighted sum of the output of the multimodal expert and the gating value as the output of the hybrid expert network;

Add the mixed output to the output of the Transformer layer of the pre-trained language sub-model.

In an optional embodiment, a preset MLP classifier is used to predict emotions, and the target model obtained includes:

The last layer of the Transformer of the pre-trained language sub-model is regarded as the final multimodal representation;

The final multimodal representation is input into the MLP classifier to predict sentiment;

Get the target model.

In an optional embodiment, the training samples are input into a target model for classification training, and the trained target model is used as a sentiment analysis model:

Training the target model by a gradient descent algorithm;

Calculating a loss function, and updating parameters according to the loss function, wherein the parameters include hybrid expert network parameters and MLP classifier parameters;

Make the number of training iterations reach the preset value and obtain the trained target model;

The trained target model is used as the sentiment analysis model.

Compared with the prior art, the multimodal sentiment analysis model construction method of the present invention has the following beneficial effects:

By inserting multimodal experts into the pre-trained language sub-model and freezing the pre-trained parameters during training, the problem of too many parameters in multimodal sentiment analysis using pre-trained language sub-models is solved. By integrating different adapters with the help of hybrid expert networks, the problem of misleading content in video and audio affecting model prediction in multimodal sentiment analysis is solved; further, by analyzing the features of different modalities at different times through multimodal routing and assigning tokens to appropriate experts for processing, compared with traditional fusion methods, the judgment of the effectiveness of multimodal features is increased, and the classification accuracy is improved.

In a second aspect, the present invention proposes a multimodal sentiment analysis model, which is constructed using the above-mentioned multimodal sentiment analysis model construction method, wherein the analysis model includes a pre-trained language sub-model, an input layer, a multimodal fusion layer, and an output layer;

The input layer and the pre-trained language sub-model are used to extract features of the input samples to obtain multimodal features; the multimodal fusion layer is used to selectively fuse the multimodal features and jointly output the output with the Transformer layer of the pre-trained language sub-model; the output layer is used to output the probability that the input samples belong to different emotion labels.

In an optional embodiment, the input layer includes an encoder, which is used to process the video data and audio data of the input sample to obtain video features and audio features; the pre-trained language sub-model is used to extract text information of the input sample to obtain text features.

In an optional embodiment, the multimodal fusion layer includes a multimodal expert and a multimodal routing, the multimodal routing is used to selectively fuse multimodal features, the multimodal expert is used to introduce multimodal features into a pre-trained language sub-model, and outputs the multimodal features together with the Transformer layer of the pre-trained language sub-model, and the last Transformer layer of the pre-trained language sub-model is used as the multimodal representation.

In a third aspect, the present invention proposes a multimodal sentiment analysis method, the analysis method comprising:

Obtain the emotion dataset to be identified;

Input samples of the emotion data set to be identified into the above multimodal emotion analysis model;

Determine the probability that samples in the emotion data set to be identified belong to different emotion labels;

The output with the highest probability is taken as the analysis result.

Compared with the prior art, the beneficial effects of the multimodal sentiment analysis model and the multimodal sentiment analysis method of the present invention are the same as the multimodal sentiment analysis model construction method described in the first aspect, so they will not be repeated here.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of a method for constructing a multimodal sentiment analysis model in an embodiment of the present invention;

FIG2 is a schematic diagram of text feature extraction of a pre-trained language sub-model in an embodiment of the present invention;

FIG3 is a schematic diagram of the principles of multimodal experts and multimodal routing in an embodiment of the present invention;

FIG4 is a flow chart of a multimodal sentiment analysis method according to an embodiment of the present invention;

FIG5 is a structural block diagram of a multimodal sentiment analysis model building device in an embodiment of the present invention.

DETAILED DESCRIPTION

In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with general skills in the technical field to which this application belongs. The words "one", "a", "the", "these" and the like in this application do not indicate a quantitative limitation, and they may be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And/or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and/or B" may mean: A exists alone, A and B exist at the same time, and B exists alone. Generally, the character "/" indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

In an embodiment of the present invention, a method for constructing a multimodal sentiment analysis model is provided. FIG. 1 is a flow chart of the method for constructing a multimodal sentiment analysis model of the present invention. As shown in FIG. 1 , the construction method includes the following steps:

S101, obtaining a pre-trained language sub-model and a training set, where the training set includes a plurality of labeled training samples;

It should be noted that in this embodiment, the pre-trained language sub-model is based on The pre-trained language sub-model of the layer Transformer can extract text features of training samples. It should be further explained that the training samples in the training set include video data, audio data and text data.

S102, extracting sample features in the training set to obtain multimodal features and aligning the multimodal features; in this embodiment, the multimodal features include video features, audio features, and text features;

Specifically, extracting sample features from the sentiment data set to obtain multimodal features includes and aligning the multimodal features includes;

Obtain text features of samples in the training set through the pre-trained language sub-model;

As shown in Figure 2 and Figure 3, the text data is fed into the word segmenter and embedding layer of the pre-trained language sub-model to obtain the text sequence ,in, is the length of the text sequence. is the embedding dimension of the pre-trained language sub-model.

The first The input and output of the Transformer layer are recorded as and ,in, satisfy

;

According to the above formula, the input of each layer of Transformer is further encoded using the pre-trained self-attention layer. Specifically, the self-attention layer uses LoRA for low-rank adaptation (LoRA is a low-rank fine-tuning technology).

Its expression is: .

The samples in the training set are encoded by a pre-trained encoder to obtain the underlying features, which include video features and audio features.

In this embodiment, a pre-trained encoder is used to encode the video data. and audio data Encode to obtain underlying features; the underlying features include video features and audio features.

in, and It is a frozen video and audio feature extraction tool. and are the extracted video and audio features, and is the feature dimension of video and audio, and is the sequence length of video and audio.

Align multimodal features;

Specifically, the video features and audio features are word-aligned, and for each word appearance time in the text feature, the corresponding video features and audio features are averaged within the time period to make the sequence lengths of the three modal features equal, that is, , which is recorded as .

S103, constructing a multimodal expert in the pre-trained language sub-model, introducing the multimodal features into the pre-trained language sub-model, and obtaining the output of the multimodal expert;

It should be noted that in order to enable the pre-trained language sub-model to perform sentiment analysis tasks, multimodal features need to be introduced into the pre-trained language sub-model. For each modality, a certain number of experts (parallel adapters) are designed to capture its feature representation at that time.

Specifically, a multimodal expert is constructed in the pre-trained language sub-model, and multimodal features are introduced into the pre-trained language sub-model. The output of the multimodal expert includes:

The multimodal expert input at the i-th word is recorded as a triple, and the multimodal expert input is obtained by calculation;

In combination with the above, in this embodiment, the multimodal experts include text experts, video experts and audio experts, and the multimodal features include text features, video features and audio features, which respectively correspond to a modality.

Therefore, the input of the multimodal expert at the i-th word (token) is recorded as a triple ( );

in, , use the following formula to get the input of text expert, video expert and audio expert;

;

Among them, {;} in the formula represents splicing.

Capturing the representation of the fusion process through multimodal experts;

First, the fusion representation after dimension reduction is obtained according to the following formula;

;

in, , is the intrinsic dimension of the adapter, is a nonlinear activation function, is the down-projection matrix, is the bias vector. The intrinsic dimensions of the adapter Much smaller than the embedding dimension of the pre-trained language sub-model , to ensure that the number of parameters in the parallel adapter is much smaller than the number of parameters in the pre-trained language sub-model.

The fused representation is mapped to the embedding dimension of the pre-trained language sub-model through the up-projection matrix to obtain the output of the multimodal expert.

Further, the projection matrix is used as follows The fusion after dimensionality reduction is represented as Mapping to the embedding dimension of the pre-trained language sub-model Dimension;

;

in, It is the output of the nth expert on modality m based on the i-th word (token). is the index of the parallel adapter, is the number of experts for modality m. is the upper projection matrix, is the bias vector, is a learnable scaling factor initialized to 1 that controls the influence of the corresponding parallel adapter.

S104, calculating the gating value of the multimodal expert, integrating the output of the multimodal expert through discrete routing, taking the weighted sum of the output of the multimodal expert and the gating value as the output of the hybrid expert network, and adding the hybrid output to the output of the Transformer layer of the pre-trained language sub-model.

It should be noted that, in order to avoid the introduction of irrelevant information, in this embodiment, multimodal features are selectively fused, a corresponding gating value is calculated for each expert, and the first K experts are selected according to the gating value.

Specifically, the gating value of the multimodal expert is calculated, the output of the multimodal expert is integrated, the weighted sum of the output of the multimodal expert and the gating value is used as the output of the hybrid expert network, and the hybrid output is added to the output of the Transformer layer of the pre-trained language sub-model, including;

Calculate the gating value of the multimodal expert through the multimodal gate;

Furthermore, the gate value is calculated by the following formula:

;

in, , is the gating vector of the i-th token.

Integrate the output of multimodal experts through discrete routing;

The first K values in the gated vector are obtained as follows:

;

in, is the gate vector The first K largest values. is the gate vector The indices of the top K values.

The weighted sum of the multimodal expert output and the gate value is used as the output of the hybrid expert network;

Specifically, the multimodal expert and discrete routing form a hybrid expert network. The first K values are normalized by softmax operation, and the weighted sum of the selected expert's output and the gated value is used as the output of the hybrid expert according to the following formula:

.

Add the mixed output to the output of the Transformer layer of the pre-trained language sub-model.

The output of the hybrid expert network is added to the output of the Transformer layer of the pre-trained language sub-model as follows:

;

Among them, FFN is a feedforward neural network module of the pre-trained language sub-model.

S105, using a preset MLP (multi-layer perceptron) classifier to predict sentiment and obtain a target model;

Specifically, the preset MLP classifier is used to predict emotions, and the target model includes;

The last layer of the Transformer of the pre-trained language sub-model is regarded as the final multimodal representation;

The final multimodal representation is input into the MLP classifier to predict sentiment, as shown in the following formula:

;

in It is the representation of the last token of the L-th layer Transformer.

Get the target model.

S106. The training samples are input into the target model for classification training, and the trained target model is used as a sentiment analysis model.

For example, given a sample label , the loss function is trained using the mean absolute error according to the following formula:

;

Specifically, the training samples are input into the target model for classification training, and the trained target model is used as the sentiment analysis model:

Train the target model through the gradient descent algorithm;

Calculate the loss function and update the parameters according to the loss function. The parameters include the hybrid expert network parameters and the MLP classifier parameters. It should be noted that the parameters also include the parameters of LORA. The target model is further optimized by updating the parameters.

Make the number of training iterations reach the preset value and obtain the trained target model;

The trained target model is used as the sentiment analysis model.

Usually, an MLP classifier is obtained after the model is constructed, or parameters of the MLP classifier are updated or trained to obtain the desired model.

By inserting multimodal experts into the pre-trained language sub-model and freezing the pre-trained parameters during training, the problem of too many parameters in multimodal sentiment analysis using pre-trained language sub-models is solved. By integrating different adapters with the help of hybrid expert networks, the problem of misleading content in video and audio affecting model prediction in multimodal sentiment analysis is solved; further, by analyzing the features of different modalities at different times through multimodal routing and assigning tokens to appropriate experts for processing, compared with traditional fusion methods, the judgment of the effectiveness of multimodal features is increased, and the classification accuracy is improved.

The embodiment of the present invention further proposes a multimodal sentiment analysis model, which is constructed using the above-mentioned multimodal sentiment analysis model construction method, and the analysis model includes a pre-trained language sub-model, an input layer, a multimodal fusion layer and an output layer;

The input layer and the pre-trained language sub-model are used to extract features from input samples to obtain multimodal features; the multimodal fusion layer is used to selectively fuse multimodal features and jointly output their output with the Transformer layer of the pre-trained language sub-model; the output layer is used to output the probability that the input samples belong to different sentiment labels.

Furthermore, the input layer includes an encoder, which is used to process the video data and audio data of the input sample to obtain video features and audio features; the pre-trained language sub-model is used to extract text information of the input sample to obtain text features.

Furthermore, the multimodal fusion layer includes a multimodal expert and a multimodal routing. The multimodal routing is used to selectively fuse multimodal features. The multimodal expert is used to introduce multimodal features into the pre-trained language sub-model and jointly output its output with the Transformer layer of the pre-trained language sub-model. The last Transformer layer of the pre-trained language sub-model is used as the multimodal representation.

The present invention extracts text features through a pre-trained language sub-model, extracts video features and audio features through an encoder, builds a multimodal expert in the pre-trained language sub-model, and integrates different adapters with the help of a hybrid expert network, thereby solving the problem of misleading content in video and audio affecting model prediction in multimodal sentiment analysis; by analyzing the features of different modalities at different times through multimodal routing, and assigning tokens to appropriate experts for processing, compared with traditional fusion methods, the judgment on the effectiveness of multimodal features is increased, and the classification accuracy is improved.

As shown in FIG4 , the embodiment of the present invention further proposes a multimodal sentiment analysis method, which includes the following steps:

S201, obtaining a dataset of emotions to be identified;

S202, inputting samples of the emotion data set to be identified into the multimodal emotion analysis model as described above;

S203, determining the probability that the samples in the emotion data set to be identified belong to different emotion labels;

S204. Output the one with the highest probability as the analysis result.

It should be noted that the samples in the emotion data set to be identified include text data, video data and audio data. By inputting the samples into the multimodal emotion analysis model, the probability of the samples belonging to different emotion labels is obtained, thereby predicting the emotion analysis results.

The embodiment of the present invention further provides a multimodal emotion model construction device, comprising:

An acquisition module 100 is used to acquire a pre-trained language sub-model and a training set, where the training set includes a plurality of labeled training samples;

The multimodal feature extraction module 200 is used to extract features of samples in the training set, obtain multimodal features and perform alignment processing on the multimodal features;

A multimodal expert construction module 300 is used to construct a multimodal expert in a pre-trained language sub-model, introduce multimodal features into the pre-trained language sub-model, and obtain the output of the multimodal expert;

A hybrid output module 400 is used to calculate the gating value of the multimodal expert, integrate the output of the multimodal expert through discrete routing, take the weighted sum of the output of the multimodal expert and the gating value as the output of the hybrid expert network, and add the hybrid output to the output of the Transformer layer of the pre-trained language sub-model;

The emotion prediction module 500 uses a preset MLP classifier to predict emotions and obtain a target model;

In the training module 600, the training samples are input into the target model for classification training, and the trained target model is used as a sentiment analysis model.

The acquisition module 100 acquires the pre-trained language sub-model and the training set, the multi-modal feature extraction module 200 extracts the sample features in the training set, and aligns the multi-modal features; the multi-modal expert construction module 300 constructs the multi-modal expert in the pre-trained language sub-model, introduces the multi-modal features into the pre-trained language sub-model, and obtains the output of the multi-modal expert; the gating value of the multi-modal expert is calculated by the mixed output module 400, the output of the multi-modal expert is integrated by discrete routing, the weighted sum of the output of the multi-modal expert and the gating value is used as the output of the mixed expert network, and the mixed output is added to the output of the Transformer layer of the pre-trained language sub-model; the emotion is predicted by the emotion prediction module 500 to obtain the target model; the target model is classified and trained by the training module 600 to obtain the emotion analysis model.

An embodiment of the present invention further provides an electronic device, the electronic device comprising a memory and a processor, the memory storing at least one computer executable instruction, the processor being configured to execute the computer executable instruction, and the computer executable instruction being executed by the processor to implement the above method embodiment, for example, the following method:

Obtain a pre-trained language sub-model and a training set, where the training set includes multiple labeled training samples;

Extract sample features from the training set, obtain multimodal features, and align the multimodal features;

Construct a multimodal expert in the pre-trained language sub-model, introduce multimodal features into the pre-trained language sub-model, and obtain the output of the multimodal expert;

Calculate the gating value of the multimodal expert, integrate the output of the multimodal expert through discrete routing, take the weighted sum of the multimodal expert output and the gating value as the output of the hybrid expert network, and add the hybrid output to the output of the Transformer layer of the pre-trained language sub-model;

Use the preset MLP classifier to predict sentiment and obtain the target model;

The training samples are input into the target model for classification training, and the trained target model is used as the sentiment analysis model.

The logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

An embodiment of the present invention further provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method provided in each of the above embodiments is implemented, for example, including:

Obtain a pre-trained language sub-model and a training set, where the training set includes multiple labeled training samples;

Extract sample features from the training set, obtain multimodal features, and align the multimodal features;

Construct a multimodal expert in the pre-trained language sub-model, introduce multimodal features into the pre-trained language sub-model, and obtain the output of the multimodal expert;

Calculate the gating value of the multimodal expert, integrate the output of the multimodal expert through discrete routing, take the weighted sum of the multimodal expert output and the gating value as the output of the hybrid expert network, and add the hybrid output to the output of the Transformer layer of the pre-trained language sub-model;

Use the preset MLP classifier to predict sentiment and obtain the target model;

The training samples are input into the target model for classification training, and the trained target model is used as the sentiment analysis model.

Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM/RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. A method for constructing a multimodal sentiment analysis model, characterized in that the construction method comprises: Obtaining a pre-trained language sub-model and a training set, wherein the training set includes a plurality of labeled training samples; Extracting features of samples in the training set to obtain multimodal features and aligning the multimodal features; Constructing a multimodal expert in the pre-trained language sub-model, introducing the multimodal features into the pre-trained language sub-model, and obtaining an output of the multimodal expert; Calculating the gating value of the multimodal expert, integrating the output of the multimodal expert through discrete routing, taking the weighted sum of the output of the multimodal expert and the gating value as the output of the hybrid expert network, and adding the hybrid output to the output of the Transformer layer of the pre-trained language sub-model; Use the preset MLP classifier to predict sentiment and obtain the target model; The training samples are input into the target model for classification training, and the trained target model is used as a sentiment analysis model. 2. The method for constructing a multimodal sentiment analysis model according to claim 1, characterized in that: Extracting sample features from the training set to obtain multimodal features and aligning the multimodal features includes: Acquire text features of samples in a training set through the pre-trained language sub-model; The samples in the training set are encoded by a pre-trained encoder to obtain underlying features, where the underlying features include video features and audio features. 3. The method for constructing a multimodal sentiment analysis model according to claim 2, characterized in that a multimodal expert is constructed in a pre-trained language sub-model, the multimodal features are introduced into the pre-trained language sub-model, and the output of the multimodal expert is obtained, comprising: The multimodal expert input at the i-th word is recorded as a triple, and the multimodal expert input is obtained by calculation; Capturing the representation of the fusion process through multimodal experts; The fused representation is mapped to the embedding dimension of the pre-trained language sub-model through the up-projection matrix to obtain the output of the multimodal expert. 4. The method for constructing a multimodal sentiment analysis model according to claim 3, characterized in that the gating value of the multimodal expert is calculated, the output of the multimodal expert is integrated, the weighted sum of the output of the multimodal expert and the gating value is used as the output of the hybrid expert network, and the hybrid output is added to the output of the Transformer layer of the pre-trained language sub-model; Calculate the gating value of the multimodal expert through the multimodal gate; Integrate the output of multimodal experts through discrete routing; Taking the weighted sum of the output of the multimodal expert and the gating value as the output of the hybrid expert network; Add the mixed output to the output of the Transformer layer of the pre-trained language sub-model. 5. The method for constructing a multimodal sentiment analysis model according to claim 4, wherein the target model is obtained by using a preset MLP classifier to predict sentiments, and the target model comprises: The last layer of the Transformer of the pre-trained language sub-model is regarded as the final multimodal representation; The final multimodal representation is input into the MLP classifier to predict sentiment; Get the target model. 6. The method for constructing a multimodal sentiment analysis model according to claim 5, characterized in that the training samples are input into a target model for classification training, and the trained target model is used as a sentiment analysis model: Training the target model by a gradient descent algorithm; Calculating a loss function, and updating parameters according to the loss function, wherein the parameters include hybrid expert network parameters and MLP classifier parameters; Make the number of training iterations reach the preset value and obtain the trained target model; The trained target model is used as the sentiment analysis model. 7. A multimodal sentiment analysis model, constructed by the multimodal sentiment analysis model construction method according to any one of claims 1 to 6, characterized in that the analysis model comprises a pre-trained language sub-model, an input layer, a multimodal fusion layer and an output layer; The input layer and the pre-trained language sub-model are used to extract features of the input samples to obtain multimodal features; the multimodal fusion layer is used to selectively fuse the multimodal features and jointly output the output with the Transformer layer of the pre-trained language sub-model; the output layer is used to output the probability that the input samples belong to different emotion labels. 8. A multimodal sentiment analysis model according to claim 7, characterized in that the input layer includes an encoder, which is used to process the video data and audio data of the input sample to obtain video features and audio features; the pre-trained language sub-model is used to extract text information of the input sample to obtain text features. 9. A multimodal sentiment analysis model according to claim 7, characterized in that the multimodal fusion layer includes a multimodal expert and a multimodal routing, the multimodal routing is used to selectively fuse multimodal features, the multimodal expert is used to introduce multimodal features into a pre-trained language sub-model, and outputs the multimodal features together with the Transformer layer of the pre-trained language sub-model, and the last Transformer layer of the pre-trained language sub-model is used as a multimodal representation. 10. A multimodal sentiment analysis method, characterized in that the analysis method comprises: Obtain the emotion dataset to be identified; Inputting a sample of the emotion data set to be identified into the multimodal emotion analysis model according to any one of claims 7 to 9; Determine the probability that samples in the emotion data set to be identified belong to different emotion labels; The output with the highest probability is taken as the analysis result.