JP7742677B2 - A method for reconstructing fine-grained tactile signals for audiovisual aids - Google Patents

Description

The present invention relates to the technical field of generating haptic signals, and in particular to a method for reconstructing fine-grained haptic signals for audiovisual aids.

As traditional multimedia application technologies mature, people's audiovisual needs are largely met, and they are beginning to pursue more dimensional and higher-level perceptual experiences. Haptic information is gradually being integrated into existing audio/video multimedia services to form multimodal services, providing a richer interactive experience. Cross-modal communication technologies have been proposed to support cross-modal services and have demonstrated some effectiveness in ensuring the quality of multimodal streams. However, applying cross-modal communication to haptic-based multimodal services still faces several technical challenges. First, haptic streams are highly sensitive to interference and noise in wireless links, resulting in degradation or even loss of haptic signals at the receiving end. This problem is particularly serious and unavoidable in remote control applications, such as remote industrial control and remote surgery. Secondly, service providers do not have tactile collection devices, but users need to sense touch. Particularly in virtual interactive application scenarios such as online immersive shopping, holographic museum guides, and virtual interactive movies, users have a very high need for tactile perception, and therefore require the ability to generate "virtual" touch sensations or tactile signals based on video and audio signals.

Currently, tactile signals that are corrupted or partially lost due to unreliable wireless communication and communication noise interference can self-recover in two ways. The first is based on traditional signal processing techniques, which use sparse representations to search for a specific signal with the most similar structure, and then use this specific signal to estimate the missing parts of the corrupted signal. The second is to mine and utilize the spatiotemporal correlations of the signal itself to achieve intra-modal self-recovery and reconstruction. However, when the tactile signal is severely corrupted or even non-existent, intra-modal reconstruction schemes fail.

In recent years, several studies have focused on the correlation between different modalities and achieved cross-modal reconstruction. In their paper "Learning cross-modal visual-tactile representation using ensembled generative adversarial networks," Li et al. proposed using image features to obtain the necessary category information, and then using this information along with noise as input to a generative adversarial network to generate a tactile spectral map of the corresponding category. However, due to the limited information obtained by mining the semantic correlation and categories between each modality, the results generated are often inaccurate. In their paper "Deep Visuo-Tactile Learning: Estimation of Tactile Properties from Images," Kuniyuki Takahashi et al. extend a single encoder-decoder network to embed both visual and tactile attributes into a latent space, focusing on the degree of tactile properties of materials represented by latent variables. Furthermore, in the paper "Teaching Cameras to Feel: Estimating Tactile Physical Properties of Surfaces from Images," Matthew Purr et al. proposed a cross-modal learning framework with adversarial learning and cross-domain associative classification to estimate tactile physical properties from a single image. While this method utilizes modal semantic information, it does not generate a complete tactile signal, making it of no practical value for cross-modal services.

The above-mentioned existing cross-modal generation methods also suffer from the following shortcomings: The training of these models all relies on large-scale training data to ensure the effectiveness of the models, and they all only use information from a single modality. However, in reality, the advantages of a single modality cannot provide a sufficient amount of information. When different modalities jointly describe the same meaning, they may contain unequal amounts of information. Complementary and enhanced information between modalities contributes to improved generation effectiveness. In practical applications, the annotation costs for large datasets are enormous, and coarse-grained, broad classification categories are often more easily obtained, while fine-grained categories are unclear. Furthermore, there is no direct correspondence between samples from different modalities, posing challenges in weak supervision and weak matching.

The technical problem that the present invention aims to solve is to provide a method for reconstructing fine-grained haptic signals for audiovisual assistance that overcomes the shortcomings of the prior art. First, cluster analysis is performed within coarse-grained categories to obtain fine-grained classifications of samples. Next, modal co-semantic constraints are performed in the fine-grained categories, with the goal of minimizing intra-category differences and maximizing inter-category differences. Finally, correlation constraints are performed by searching for audiovisual samples that positively match the haptic signal in the fine-grained subcategories, thereby achieving high-quality, fine-grained reconstruction of the haptic signal.

In order to solve the above technical problems, the present invention provides a method for reconstructing fine-grained tactile signals for audiovisual assistance,
inputting a haptic signal into a haptic autoencoder and extracting features from the haptic signal through a clustering task;
transferring and optimizing the feature extraction ability of the haptic autoencoder to the audio feature extraction network and the image feature extraction network by a cross-modal transfer learning method;
Constraining the extracted tactile, audio, and image modal features by jointly considering clustering constraints, centrality constraints, and sorting constraints to bring modal features belonging to the same meaning closer together but separate modal features that do not belong to the same meaning, thereby obtaining tactile, audio, and image modal features with fine-grained classification;
Constructing a triplet set based on tactile, audio and image modal features, performing shared semantic learning of triplet constraints, optimizing a multimodal fusion mapping function, and obtaining fusion features containing shared semantic information;
and preconfiguring a haptic generation network and inputting the fused features into the haptic generation network to reconstruct a haptic signal.

Furthermore, the step of preconfiguring a haptic generation network and inputting the fused features into the haptic generation network to reconstruct a haptic signal includes:
Presetting parameters of a haptic autoencoder, an audio feature extraction network, and an image feature extraction network;
Presetting parameters of a multimodal fusion mapping function and parameters of a haptic generation network;
training a haptic autoencoder, an audio feature extraction network, an image feature extraction network, a multimodal fusion mapping function, and a haptic generation network;
The method includes inputting the just-received image signal and audio signal into a trained image feature extraction network and an audio feature network, respectively, to obtain image features and audio features, then inputting the features into a multimodal fusion mapping function to obtain fusion features, and finally inputting the fusion features into a trained haptic generation network to obtain a reconstructed haptic signal.

Furthermore, inputting tactile signals into a tactile autoencoder and extracting features from the tactile signals through a clustering task is
The haptic signal is input to a haptic autoencoder for learning, and the corresponding haptic features are extracted. Then, the haptic signal is clustered based on the K-means algorithm according to the haptic features, i.e.,
h is the input haptic signal, h = {h _i } _{i = 1, ..., N} , where i indicates the sort subscript of the input haptic signal, and N is the total amount of the input haptic signal. After passing through the encoding module E _h (·) of the autoencoder, f _i ^h = E _h (h _i ; θ _he ) is the feature representation of the haptic signal h _i , where f ^h = {f _i ^h } _{i = 1, ..., N} , where θ _he is the parameter of the encoding module. f _i ^h is input to the decoding module D _h (·), and the output haptic signal
where θ _hd is the parameter of the decoding module, and performs clustering based on the K-means algorithm on the features f _i ^h to output the corresponding category tags s _i ^h . The parameters in the above process are jointly estimated, and the loss function
(however,
is the reconstruction error of the encoder, N is the number of haptic signals,
is the clustering error of K-means, M is the cluster center vector matrix for obtaining tactile data using the K-means algorithm, m _c in the c-th column of the M matrix indicates the center of mass of the c-th cluster, θ _h = [θ _he , θ _hd ] is the parameter of the encoder module and the decoder module, s _j,i ^h is the j-th element of s _i ^h , and if the element s _j,i ^h is 1 and the other elements are all 0, it indicates that the original tactile signal h _i corresponding to s _i ^h belongs to the j-th category, and l is the least squares loss.
where λ is a regularization parameter, λ≧0;
Minimizing L _clu ^h to estimate θ _h and obtain f _i ^h and s _i ^h .

Furthermore, transferring the feature extraction ability of a haptic autoencoder to an audio feature extraction network and an image feature extraction network through a cross-modal transfer learning method is
The feature self-adaptation method minimizes the maximum average difference criterion between the tactile and audiovisual areas to achieve the transition, i.e.,
Let the distributions of the haptic, audio, and image signal sets be P, Q, and R, respectively, denote the MMD between the haptic and audio signals as MMD _k (P,Q), the MMD between the haptic and visual signals as MMD _k (P,R), and in the reproduction kernel Hilbert space H _k , H _k contains a function set f defined on a non-empty set, and the square of the MMD is
and
where haptic, audio, and image signals are passed through respective feature extraction networks φ to obtain extracted feature vectors, the haptic feature vector is denoted as ^φh (h; _θhe ), i.e., it is the output of the encoding module of the autoencoder, the audio and image feature vectors are ^φa (a; _θa )= ^fa , ^φv (v; _θv )= ^fv , the three modal feature set is denoted as ( ^fh , ^fa , ^fv ), _θa and _θv are the parameters of the audio and image feature extraction networks respectively, _θhe is the parameter of the encoding module, and for any function f∈Hk and any _X∈P ,
where μ _k (P) is the mean embedding of P in H _k , i.e., one element representation in H _k space of the distribution P, f(X) denotes the mapping of X to H _k space by function f, <·,·> _{H k} is the dot product operation, and similarly,
and μ _k (Q) is the mean embedding of Q in H _k ,
and μ _k (R) is the mean embedding of R in H _k ;
handle
The value of is calculated as the loss function of cross-modal transfer, and the specific formula is
And,
By optimizing the _LCT , we guide the information flow between the haptic feature extraction autoencoder model and the audio-visual feature extraction network, effectively transferring the feature extraction capabilities of the autoencoder for the haptic modality to the audio-visual feature extraction network, i.e., estimating θ _a and θ _v .

Further optimizing the audio feature extraction network and the image feature extraction network may include:
The classification loss function is
(where s _i ^a and s _i ^v are the category tags of the audio signal and the image signal, respectively; by minimizing L _clu ^av , the optimal values of θ _a and θ _v are further obtained; p(f _i ^a ; θ _a ) means the probability of obtaining the audio signal category s _i ^a when the audio feature extraction network input is f _i ^a and the network parameter is θ _a ; p(f _i ^v ; θ _v ) means the probability of obtaining the image signal category s _i ^v when the image feature extraction network input is f _i ^v and the network parameter is θ _v ; f ^a = {f _i ^a } _{i = 1, ... , N} , f ^v = {f _i ^v } _{i = 1, ... , N} ); and L _clu ^h , L _CT and L _clu ^av
Combined,
Obtaining the overall objective function L _clu = L _clu ^h + L _CT + L _clu ^av ; and by minimizing L _clu , the optimal θ _h , θ _a , and θ _v can be obtained, and after determining the parameters, the features f ^h , ^fa , f ^v corresponding to the tactile signal, audio signal, and image signal can be obtained.

Furthermore, by constraining the extracted tactile, audio, and image modal features by jointly considering the clustering constraint, the centrality constraint, and the sorting constraint, the modal features belonging to the same meaning are brought close to each other, but the modal features not belonging to the same meaning are separated, thereby obtaining tactile, audio, and image modal features with fine-grained classification.
To ensure compactness between features of the same fine-grained subcategory, clustering learning under center constraints is performed on three types of modal signals. To achieve better fine-grained classification performance, features of the same subcategory should be adjacent in a common space, with the goal of minimizing intra-category variance. Clustering learning is driven by minimizing the distance from a feature to its subcategory center. The subcategory center of the tactile signal is set as the common subcategory center of the cross-modal signal to ensure compactness between semantic features of the same category of cross-modal signals. The loss function of clustering learning under center constraints is:
is defined as
where N is the number of haptic, audio, and image signals, s _i ^h , s _i ^a , and s _i ^v indicate the categories of haptic signals, audio signals, and image signals, respectively, the audio signals and image signals share the cluster center vector matrix M of the haptic signals, and the above process includes bringing each modal feature f _i ^a , f _i ^v , and f _i ^h with similar meaning closer to each other.

Furthermore, to ensure that the features of different fine-grained subcategories have a certain sparsity, we perform clustering learning under sorting constraints on the three types of modal signals. The goal of the centrality constraint is to minimize the intra-category variance, while the goal of the sorting constraint is to maximize the inter-category variance, so that the feature outputs of different subcategories are less similar than the feature outputs of the same subcategory. The sorting constraint is:
is defined as
where C is the total number of categories after the tactile signals are clustered using the K-means algorithm, and the above process brings modal features f _i ^a , f _i ^v , and f _i ^h with similar meanings closer together, while separating modal features with different meanings as much as possible.

Furthermore, constructing a triplet set based on the modal features of tactile, audio, and image, conducting shared semantic learning of triplet constraints, optimizing the multimodal fusion mapping function, and obtaining fusion features containing shared semantic information is
Randomly selecting one haptic signal sample h _i from a fine-grained subcategory and setting the sample as an anchor;
Selecting a sample v _i ⁺ from the image dataset that belongs to the same category as h _i and whose semantic features are closest to f _i ^h as a positive match sample, and selecting a sample v _j ⁻ that does not belong to the same category as h _i and whose semantic features are closest to f _i ^h as a negative match sample;
thereby constructing a set of triplets {(h _i ,v _i ⁺ ,v _j ⁻ )} for the samples in the dataset;
Similarly, obtaining a set of triplets {(h _i , a _i ⁺ , a _j ⁻ )} consisting of haptic signal samples and audio signal samples;
Semantic features f _i ^h of anchor point h _i and positive match semantic features in the corresponding subclass in the audio-visual modal
and minimize the distance between f _i ^h and the semantic feature of negative match
By maximizing the semantic feature between and and having a minimum interval δ, the two triplet loss functions obtained are
and
We introduce an integration paradigm to achieve high-level fusion of multimodal features, i.e.,
By fusing f ^a and f ^v , the process becomes
f ^m =F _m (f ^a ,f ^v ;θ _m )
where f ^m is the output of multimodal fusion in the shared semantic subspace, i.e., the fusion feature, F _m (·) is the multimodal fusion mapping function with parameter θ _m , and F _m (·) takes a linear weighting of f ^a and f ^v ;
fusing f ₊ ^a and f ₊ ^v into f ₊ ^m , and f _- ^a and f _- ^v into f _- ^m ;
The triplet loss is used to constrain the fused features, i.e.,
and
The objective function of shared semantic learning is modeled by combining three loss functions.
It can be shown that L _syn =L _syn ^a +L _syn ^v +L _syn ^m ;
and minimizing L _Syn to obtain the optimal θ _m .

Furthermore, a haptic generation network is constructed, that is, a haptic generation network G(·) is constructed, and its structure is the same as D _h (·), and its network parameter θ _hd is set as the initial value of the parameter θ _G of G(·);
The fused features containing the required semantic information are input into a haptic generation network G(·) to obtain the desired haptic signal h′, and the generated haptic signal h′ is remapped to a haptic feature f ^h′ by E _h (·), and a category center is selected to perform semantic constraints on the haptic feature f ^h′ , so that the final loss function is
is shown as
however,
and
indicates the similarity between features f ^h and f ^h′ ,
is the clustering loss of f ^h′ , and these are used together as the regularization terms of the loss function. By optimizing the loss function, the optimal value of θ _G is obtained, i.e., G(·) is determined.

Furthermore, we pre-set the parameters of the haptic autoencoder, audio feature extraction network, and image feature extraction network.
Presetting the parameters of the multimodal fusion mapping function and the parameters of the haptic generation network;
Training the haptic autoencoder, the audio feature extraction network, the image feature extraction network, the multimodal fusion mapping function, and the haptic generation network includes step 1 and step 2;
In the step 1, θ _v , θ _a , θ _he , θ _hd , and M are preset, and {s _i ^h }, {s _i ^a }, and {s _i ^v } are optimized.
Step 11: initializing network parameters θ _v , θ _a , θ _he , θ _hd , node system tags {s _i ^h }, { ^{s ia} _} , {s _i ^v }, and category center matrix M, and setting the number of clusters C, learning rate μ ₁ , and number of iterations T;
{s _i ^h }, {s _i ^a }, {s _i ^v }, and M are fixed, and θ _v , θ _a , θ _he , and θ _hd are optimized based on stochastic gradient descent, i.e.,
and
where step 12 is where ∇ is the partial derivative of each loss function;
Fix θ _v , θ _a , θ _he , θ _hd , and M, and optimize {s _i ^h }, {s _i ^a }, {s _i ^v }, i.e.,
Step 13, where
Fix θ _v , θ _a , θ _he , θ _hd , and {s _i ^h }, {s _i ^a }, {s _i ^v } and optimize M, i.e.,
Step 14, where
If t<T, jump to step 412; if t=t+1, continue with the next iteration; otherwise, end the iteration;
and step 16, after T iterations, obtaining optimal audio feature extraction network parameters θ _a , image feature extraction network parameters θ _v , haptic autoencoder parameters θ _he , θ _hd , node system tags {s _i ^h }, {s _{i a} ^} , {s _i ^v }, and a cluster center vector matrix M of the haptic data;
In the step 2, θ _m and θ _G are estimated based on a stochastic gradient descent method, and the step 2 includes the following steps:
Step 21 of initializing θ _m , learning rates μ ₂ , μ ₃ , and number of iterations n ₁ ;
Based on L _Syn, θ _m is estimated using stochastic gradient descent, i.e.,
Step 22, where
Based on L _Gen , θ _G is updated using stochastic gradient descent, i.e.,
Step 23, where
If n< _n1 , jump to step 22; if n=n+1, continue with the next iteration; otherwise, step 24 to end the iteration;
and step 25 of obtaining the optimal θ _m and θ _G after one _n iteration.

Compared to the prior art, the present invention has the following technical advantages by using the above technical solutions:

This invention uses a deep clustering algorithm for cross-modal transfer to learn fine-grained classification of three types of modal samples, then performs shared semantic learning, fully utilizing the advantages of multi-modal feature fusion, ultimately realizing the generation of fine-grained haptic signals based on clustering constraints, making full use of existing datasets with weak supervision and weak matching problems, thereby generating high-quality fine-grained haptic signals that better meet the requirements of cross-modal services.

1 is a flowchart of a method for reconstructing fine-grained haptic signals for audiovisual aids according to the present invention; 1 is a structural schematic diagram of a complete network according to the present invention; FIG. 1 is a schematic diagram of the architecture of a deep clustering model based on cross-modal transfer according to the present invention. 10A and 10B show the reconstruction results of tactile signals of the present invention and other comparative methods.

To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be described in detail below with reference to the drawings and specific embodiments.

The present invention provides a method for reconstructing fine-grained tactile signals for audiovisual assistance, the flowchart of which is shown in Figure 1, and the method includes the following steps 1 to 5.

In step 1, haptic signals are first input into a haptic autoencoder, and feature extraction of the haptic signals is achieved through clustering. Next, cross-modal transfer learning technology is used to transfer and optimize the feature extraction capabilities of the haptic autoencoder to an audio feature extraction network and an image feature extraction network, respectively. The extracted haptic, audio, and image modal features are then further constrained by jointly considering clustering constraints, centrality constraints, and sorting constraints, thereby approximating modal features that belong to the same meaning and separating modal features that do not belong to the same meaning, thereby obtaining haptic, audio, and image modal features with fine-grained classification.

(1-1) First, clustering-constrained feature learning is performed on three types of modal signals: haptic, image, and audio, to obtain segmental features with fine-grained subcategories. This can be divided into the following three steps: Step (1-1-1) to Step (1-1-3).

(1-1-1) In the first step, the tactile signal is input to an autoencoder for learning, and the corresponding tactile features are extracted. Then, the tactile signal is clustered based on the K-means algorithm according to the tactile features, i.e.,
Specifically, suppose h _i is the input haptic signal, h = {h _i } _{i = 1, ..., N,} where i indicates the sort subscript of the input haptic signal, and N is the total amount of the input haptic signal. After passing through the encoding module E _h (·) of the autoencoder, f _i ^h = E _h (h _i ; θ _he ) is the feature representation of the haptic signal h _i , where f ^h = {f _i ^h } _{i = 1, ..., N} , and θ _he is the parameter of the encoding module. Then, f _i ^h is input to the decoding module D _h (·), and the output haptic signal
where θ _hd is the parameter of the decoding module, and performs clustering based on the K-means algorithm on the features f _i ^h to output the corresponding category tags s _i ^h . The parameters in the above process are jointly estimated, and the loss function
(however,
is the reconstruction error of the encoder, N is the number of haptic signals,
is the clustering error of K-means, M is the cluster center vector matrix for obtaining tactile data using the K-means algorithm, m _c in the c-th column of the M matrix indicates the center of mass of the c-th cluster, θ _h = [θ _he , θ _hd ] is the parameter of the encoder module and the decoder module, s _j,i ^h is the j-th element of s _i ^h , and if the s _j,i ^h value of the element is 1 and the other elements are all 0, it indicates that the original tactile signal h _i corresponding to s _i ^h belongs to the j-th category, and l is the least squares loss.
where λ≧0 is a regularization parameter),
By minimizing L _clu ^h , θ _h can be estimated and f _i ^h and s _i ^h can be obtained.

(1-1-2) In the second step, the acquired tactile features are transferred to the feature extraction process of the image and audio signals using cross-modal transfer learning technology. That is, the transfer is achieved by minimizing the maximum mean difference (MMD) criterion between the tactile and audiovisual domains using a feature self-adaptation method.

Specifically, let the distributions of the haptic, audio, and image signal sets be P, Q, and R, respectively. The MMD between the haptic signal and the audio signal is denoted as _MMDk (P,Q), and the MMD between the haptic signal and the visual signal is _MMDk (P,R). In the reproduction kernel Hibert space _Hk , _Hk contains a function set f defined on a non-empty set, and the square of the MMD is
and
where haptic, audio, and image signals are passed through respective feature extraction networks φ to obtain extracted feature vectors, and the haptic feature vector may be denoted as ^φh (h; _θhe ), i.e., the output of the encoding module of the autoencoder in the previous step, the audio and image feature vectors are ^φa (a; _θa )= ^fa , ^φv (v; _θv )= ^fv , and the three-modal feature set may be denoted as ( ^fh , ^fa , ^fv ). _θa and _θv are the parameters of the audio and image feature extraction networks, respectively, and _θhe is the parameter of the encoding module. For any function _f∈Hk and any X∈P,
where μ _k (P) is the mean embedding of P in H _k , i.e., one element representation in H _k space of the distribution P, f(X) denotes the mapping of X to H _k space by function f, <·,·> _{H k} is the dot product operation, and similarly,
and μ _k (Q) is the mean embedding of Q in H _k ,
and μ _k (R) is the mean embedding of R in H _k .

handle
The value of is calculated as the loss function of cross-modal transfer, and the specific formula is
is.

By optimizing _LCT , we can guide the information flow between the haptic feature extraction autoencoder model and the audio-visual feature extraction network, thereby effectively transferring the feature extraction capabilities of the autoencoder for the haptic modality to the audio-visual feature extraction network, i.e., estimating θ _a and θ _v .

(1-1-3) In the third step, the audio and image feature extraction network is further optimized. Here, the video feature extraction network uses a VGG network design style, i.e., a 3x3 convolutional filter and a 2x2 max pooling layer with a step width of 2 and no padding. The network is divided into four blocks, each containing two convolutional layers and one pooling layer, with the number of filters doubling between successive blocks. Finally, max pooling is performed at all spatial locations to generate a single 512-dimensional semantic feature vector. This semantic feature vector is then input to a three-layer fully connected neural network (256-128-32) with K nodes and one fully connected layer with a softmax function. The 32-dimensional vector is the visual signal feature vector, which is then further trained through subsequent shared semantic learning based on triplet constraints. K is the number of fine-grained subcategories in each coarse-grained category, and K is 3 in this experimental dataset. The associated network architecture for audio signals is similar in configuration to visual signals and shares classifiers with visual signals.

The designed classification loss function is
(where s _i ^a and s _i ^v are the category tags of the audio signal and the image signal, respectively; by minimizing L _clu ^av , the optimal values of θ _a and θ _v can further be obtained; p(f _i ^a ; θ _a ) means the probability of obtaining the audio signal category s _i ^a when the audio feature extraction network input is f _i ^a and the network parameter is θ _a ; and p(f _i ^v ; θ _v ) means the probability of obtaining the image signal category s _i ^v when the image feature extraction network input is f _i ^v and the network parameter is θ _v ).

Notably, the tags are obtained using the following incremental policy: First, pseudo-tags are added to the data, and then these data are used to optimize the model, enhancing the model's ability to classify audiovisual signals. Then, the trained model is used to perform pseudo-tag operations, thereby updating the pseudo-tags. This incremental optimization method gradually improves the network's fine-grained classification ability.

In short, the total objective function in step (1-1) is a combination of the above three loss functions,
It may be shown as L _clu =L _clu ^h +L _CT +L _clu ^av ,
By minimizing L _clu , the optimal θ _h , θ _a , and θ _v can be obtained, and after determining the parameters, the features f ^h , f ^a , and f ^v corresponding to the haptic signal, audio signal, and image signal can be obtained;
(1-2) To ensure compactness between features of the same fine-grained subcategory, clustering learning under center constraints is performed on three types of modal signals. To achieve better fine-grained classification performance, features of the same subcategory should be adjacent in a common space, with the goal of minimizing intra-category variance. Specifically, clustering learning is driven by minimizing the distance from a feature to its subcategory center. By setting the subcategory center of the tactile signal as the common subcategory center of the cross-modal signal, compactness between semantic features of cross-modal signals of the same category is ensured. The loss function for clustering learning under center constraints is:
is defined as
where N is the number of haptic, audio, and image signals, s _i ^h , s _i ^a , and s _i ^v respectively indicate the categories of haptic signals, audio signals, and image signals, and audio data and image data share the cluster center vector matrix M of haptic data. Minimizing L _cen can effectively solve the problem of large intra-category discrepancies in fine-grained classification. Through the above process, modal features f _i ^a , f _i ^v , and f _i ^h with similar meanings are made closer to each other.

(1-3) To ensure that the features of different fine-grained subcategories have a certain sparsity, we perform clustering learning under sorting constraints on three types of modal data. The goal of the centrality constraint is to minimize the within-category variance, while the goal of the sorting constraint is to maximize the between-category variance, so that the feature outputs of different subcategories are less similar than the feature outputs of the same subcategory. The sorting constraint is:
is defined as
where C is the total number of categories, i.e., the number of clusters, after the tactile signals are clustered using the K-means algorithm. Minimizing L _rank can effectively solve the problem of small inter-category differences in fine-grained classification. Through the above process, modal features f _i ^a , f _i ^v , and f _i ^h with similar meanings are brought closer together, while modal features with different meanings are separated as much as possible.

In step 2, after obtaining tactile, audio, and image features with fine-grained classification, the three modal features are constructed into a triplet set, triplet-constrained shared semantic learning is performed, a multimodal fusion mapping function is optimized, and fusion features containing shared semantic information are obtained, laying the foundation for generating tactile signals.

Step 2 specifically includes the following:

(2-1), one haptic signal sample h _i is randomly selected from a certain fine-grained subcategory and this sample is designated as the anchor. Next, a sample v _i ⁺ belonging to the same category as h _i and whose semantic features are closest to f _i ^h is selected from the image dataset as a positive match sample. After that, a sample v _j ⁻ not belonging to the same category as h _i and whose semantic features are closest to f _i ^h is selected as a negative match sample. This constructs a triplet set {(h _i , v _i ⁺ , v _j ⁻ )} for all samples in the dataset. Similarly, a triplet set {(h _i , a _i ⁺ , a _j ⁻ )} consisting of haptic signal samples and audio signal samples can be obtained. The semantic features f _{i h} of the anchor point h ⁱ and the positive match semantic features in the corresponding subcategory in the audio-image modal are
and minimize the distance between f _i ^h and the semantic feature of negative match
and there is one minimum interval δ, where δ is 1. The two triplet loss functions obtained are
is.

(2-2) Based on the above, we introduce an integration paradigm to achieve advanced multimodal feature fusion. Specifically, audiovisual data first passes through an audio feature extraction network and an image feature extraction network to obtain features f ^a and f ^v , respectively, and then f ^a and f ^v are fused. The process is as follows:
f ^m =F _m (f ^a ,f ^v ;θ _m )
(where f ^m is the output of multimodal fusion in the shared semantic subspace, i.e., the fusion feature, and F _m (·) is a mapping function with parameter θ _m ; in general, F _m (·) takes a linear weighting of f ^a and f ^v ).

We fuse f ₊ ^a and f ₊ ^v into f ₊ ^m , and f ₋ ^a and f ₋ ^v into f ₋ ^m . Similarly, we use triplet loss to constrain the fused features, i.e.,
is.

(2-3) The objective function of shared semantic learning may be modeled by combining three loss functions:
It may be shown as L _syn =L _syn ^a +L _syn ^v +L _syn ^m .

By minimizing L _Syn , the optimal θ _m is obtained, which lays the foundation for the next stage of haptic signal generation.

In step 3, the fused features are input into a haptic generation network to generate the desired haptic signal h'.

Step 3 specifically includes the following:

First, a haptic generation network is constructed. Since the haptic decoder D _h (·) is trained as part of the complete autoencoder in step (1), here we construct another haptic generation network G(·), whose structure is the same as D _h (·) (32-128-256-Z), and whose network parameters θ _hd are the initial values of the parameters θ _G of G(·). The fused features containing the required semantic information are input into the haptic generation network G(·) to obtain the desired haptic signal h′, and the generated haptic signal h′ is remapped to a 32-dimensional haptic feature f ^h′ by E _h (·), and a category center is selected to impose semantic constraints on the haptic feature f ^h′ . Obviously, the distance between the haptic feature f ^h′ and the category center of the corresponding category should be as small as possible, but the distance between the haptic feature f h′ and other category centers should be as large as possible. The final loss function is
may be shown as
however,
and
indicates the similarity between features f ^h and f ^h′ ,
is the clustering loss of f ^h′ , and they are jointly used as the regularization term of the loss function. By optimizing the loss function, the optimal value of θ _G can be obtained, i.e., G(·) can be determined.

In step 4, the model is trained. That is, model training is divided into two steps. In the first step, parameters in the haptic autoencoder, audio feature extraction network, and image feature extraction network are estimated. In the second step, parameters of the multimodal fusion mapping function and the haptic generation network are estimated. Through these steps, the haptic autoencoder, audio feature extraction network, image feature extraction network, multimodal fusion mapping function, and haptic generation network are trained.

Specifically, Step 4 is as follows:

Estimation of θ _v , θ _a , θ _he , θ _hd , and M is completed, and {s _i ^h }, {s _i ^a }, {s _i ^v } are optimized.

In step 411, the network parameters θ _v , θ _a , θ _he , θ _hd , the node system tags {s _i ^h }, {s _i ^a }, {s _i ^v }, and the category center matrix M are initialized, the number of clusters C is set, the learning rate μ ₁ = 0.0001, and the number of iterations T = 600.

Step 412, {s _i ^h }, {s _i ^a }, {s _i ^v }, and M are fixed, and θ _v , θ _a , θ _he , θ _hd are optimized, i.e.,
and
where ∇ is the partial derivative of each loss function.

Step 413, fix θ _v , θ _a , θ _he , θ _hd , and M, and optimize {s _i ^h }, {s _i ^a }, {s _i ^v }, i.e.,
is.

Step 414: Fix θ _v , θ _a , θ _he , θ _hd , and {s _i ^h }, {s _i ^a }, {s _i ^v } and optimize M, i.e.,
is.

In step 415, if t < T, jump to step 412; if t = t + 1, continue with the next iteration; otherwise, end the iteration.

In step 416, after T iterations, the optimal audio feature extraction network parameters θ _a , image feature extraction network parameters θ _v , haptic autoencoder parameters θ _he and θ _hd , node system tags {s _i ^h }, {s _i ^a }, {s _i ^v }, and cluster center vector matrix M of the haptic data are obtained.

In particular, when updating m _k
where c _k ⁱ is the set of indexes assigned to cluster k from the first sample to the current sample, but the historical data that has already appeared is insufficient to represent the overall cluster structure, and s _i ^h may be incorrect. Therefore, this algorithm assumes that the number of data samples contained in each cluster is approximately balanced, and based on this, the gradient update step is designed to update m _k , and 1/c _k ⁱ is used to control the learning rate, where c _k ⁱ is the number of times a sample is assigned to cluster k before the i-th sample is processed by the algorithm. In this way, the update of M may be regarded as an SGD step.

(4-2) Based on SGD, the estimation of θ _m and θ _G is completed.

In step 421, θ _m is initialized, the batch size bactch=64, the learning rates μ ₂ , μ ₃ =0.0001, and the number of iterations n ₁ =600.

Step 422: Fine-tune θ _m using stochastic gradient descent based on L _Syn , i.e.,
is.

Step 423: Update θ _G using stochastic gradient descent based on L _Gen , i.e.,
is.

Step 424, if n< _n1 , jump to step 422, if n=n+1, continue next iteration, otherwise end iteration.

Step 425: After 600 iterations, the optimal n = n + 1 is obtained.

In step 5, the just-received image signal and audio signal are input into the trained image feature extraction network and audio feature network, respectively, to obtain image features and audio features, respectively; then, the above features are input into the multimodal fusion mapping function to obtain fusion features; and finally, the fusion features are input into the trained haptic generation network to obtain a reconstructed haptic signal.

Step 5 specifically includes the following:

The just-received image signal v and audio signal a are input to the trained image feature extraction network and audio feature extraction network, respectively, and the image features are extracted.
and input the trained multimodal fusion mapping function to obtain the fusion feature
Get
is input to the trained G(·), and the final reconstructed tactile signal is
Get.

As can be seen from the experimental results below, compared to conventional methods, our invention achieves tactile signal synthesis through the complementary fusion of multimodal meanings, achieving a higher generation effect.

This example uses the LMT cross-modal dataset proposed in the paper "Multimodal feature-based surface material classification," which includes samples from nine semantic categories: grid, stone, metal, wood, rubber, fiber, foam, foil, paper, textiles, and fabrics. This example selects five major categories (each of which contains three subcategories) for the experiment. The LMT dataset is reconstructed by first referencing the training and test sets for each material example, and obtaining 20 image samples, 20 audio signal samples, and 20 tactile signal samples for each example. Next, the neural network is trained by augmenting the data. Specifically, each image is flipped horizontally and vertically, rotated at an arbitrary angle, and, in addition to conventional methods, techniques such as random scaling, cutting, and offsetting are used. With this, we expand the data of each category to 100, so there are a total of 1500 images with a size of 224 ^* 224. In the dataset, 80% is selected to be used for training, while the remaining 20% is used for testing and performance evaluation. The experiment is initially set up so that the fine-grained categories are unknown.

(1) Clustering Results To verify the effectiveness of the clustering method of the present invention, the clustering method was compared with several baseline methods, including:

K-means (KM)
The K-Means algorithm clusters the samples of image, audio, and haptic modal, respectively.

Autoencoder + K-means (AE+KM, Autoencoder followed by K-means)
This is a two-step method: first, we obtain feature representations for each modality by reconstructing and learning signal samples of different modalities, and then cluster them using K-means.

Triple deep clustering model (3-DCN, 3-Deep Clustering Network)
The DCN model is used to cluster signals of different modalities.

The present invention uses the method of this example.

When presenting the results, we select average values across multiple subcategories and mainly use three indices: normalized mutual information (NMI), adjusted Rand index (ARI), and clustering accuracy (ACC). The experimental results are shown in Table 1.

Table 1 shows the results of applying the present invention, 3-DCN, AE+KM, and KM to the LMT dataset. As can be seen, the method of the present invention is highly competitive on this dataset, with results significantly better than those of traditional clustering algorithms and common deep clustering algorithms. Analysis suggests that this may be due to the fact that the application scenarios of other algorithms are all unimodal, which can easily lead to imbalances in clustering results. Theoretically, the number of samples in a given category of coexisting cross-modal data should be equal. Furthermore, the subcategory features learned by this clustering method are significantly more compact and more distinctive between different categories, which contributes to the subsequent reconstruction of tactile signals.

(2) Tactile Reconstruction Results After determining the fine-grained category, we compare the proposed fine-grained tactile reconstruction method with the following several methods.

Existing Method 1
The ensemble generative adversarial networks (E-GANs) in the paper "Learning cross-modal visual-tactile representation using ensemble generative adversarial networks" (authors X. Li, H. Liu, J. Zhou, and F. Sun) utilize image features to obtain the necessary category information. Then, the category information, along with noise, is input to the generative adversarial network to generate a tactile spectral map of the corresponding category, which is then converted into a tactile signal.

Existing Method 2
The deep visio-tactile learning method (DVTL, Deep Visio-Tactile Learning) in the paper "Deep Visuo-Tactile Learning: Estimation of Tactile Properties from Images" (authors: Kuniyuki Takahashi and Jethro Tan) extends the traditional encoder-decoder network with latent variables to embed visual and tactile attributes into the latent space.

Existing Method 3
The paper "Teaching Cameras to Feel: Estimating Tactile Physical Properties of Surfaces from Images" (authors: Matthew Purri and Kristin Dana) proposes a joint-encoding-classification-generative network (JEC-GAN), which uses different encoding networks to encode each modal instance into a shared internal space, and then couples the embedded visual and tactile samples together in the latent space using paired constraints. Finally, using visual information as input, the corresponding tactile signal is reconstructed by a generative network.

This experiment is analyzed from two perspectives: quantitative and qualitative. First, Table 2 shows the tactile signal reconstruction performance of each method from multiple perspectives: root mean square error (RMSE), structural similarity (SIM), and classification accuracy (ACC). Table 2 shows the experimental results of the present invention.

As can be seen from Table 2 and Figure 4, the method of the present invention has clear advantages over the most advanced methods mentioned above. The reasons are as follows: (1) The tactile signal reconstruction method of the present invention uses a cross-modal clustering algorithm to clarify fine-grained subcategories and utilizes centrality and sorting constraints to effectively improve the compactness and distinctiveness of semantic features. (2) When the correspondence between visual, auditory, and tactile samples is specified by humans, it is relatively subjective and therefore not accurate enough. In contrast, in the fine-grained tactile signal reconstruction method for audiovisual assistance of the present invention, during training, the model self-selects the audiovisual semantic features that are closest to the tactile semantic features as the input of its generative network.

In another embodiment, the haptic encoder in step 1 of the present invention may be replaced by a one-dimensional convolutional neural network (1D-CNN) using a feedforward neural network.

The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited to this. Any modifications or substitutions that a person skilled in the art can easily make within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention.

Claims (10)

1. A method for reconstructing fine-grained haptic signals for audiovisual aids, comprising:
inputting a haptic signal into a haptic autoencoder and extracting features from the haptic signal through a clustering task;
Transferring and optimizing the feature extraction ability of the haptic autoencoder to the audio feature extraction network and the image feature extraction network by a cross-modal transfer learning method , i.e., optimizing the parameters of the audio feature extraction network and the image feature extraction network by minimizing the cross-modal transfer loss function and the classification loss function derived from the maximum mean difference (MMD) criterion between the haptic domain and the audiovisual domain;
A step of performing clustering constraint feature learning and clustering learning under center constraint and sorting constraint for three types of modal signals of tactile, audio, and image , thereby approximating tactile, audio, and image modal features that belong to the same meaning but separating modal features that do not belong to the same meaning, and obtaining tactile, audio, and image modal features with fine-grained classification;
Constructing a triplet set based on tactile, audio, and image modal features, and performing triplet-constrained shared semantic learning, i.e., optimizing parameters of a multimodal fusion mapping function by modeling and minimizing the objective function of shared semantic learning, and inputting the audio features and image features into the optimized multimodal fusion mapping function to obtain fusion features containing shared semantic information;
a step of presetting a haptic generation network and inputting the fused features into the haptic generation network to reconstruct a haptic signal. The step of preconfiguring a haptic generative network and inputting the fused features into the haptic generative network to reconstruct a haptic signal includes:
Presetting parameters of a haptic autoencoder, an audio feature extraction network, and an image feature extraction network;
Presetting parameters of a multimodal fusion mapping function and parameters of a haptic generation network;
training a haptic autoencoder, an audio feature extraction network, an image feature extraction network, a multimodal fusion mapping function, and a haptic generation network;
2. The method for reconstructing fine-grained haptic signals for audiovisual assistance according to claim 1, comprising: inputting the just-received image signal and audio signal into a trained image feature extraction network and an audio feature network, respectively, to obtain image features and audio features, respectively; then inputting the features into a multimodal fusion mapping function to obtain fusion features; and finally inputting the fusion features into a trained haptic generation network to obtain a reconstructed haptic signal. Inputting tactile signals into a tactile autoencoder and extracting features from the tactile signals through a clustering task is
The haptic signal is input to a haptic autoencoder for learning, and the corresponding haptic features are extracted. Then, the haptic signal is clustered based on the K-means algorithm according to the haptic features, i.e.,
h is the input haptic signal, h = {h _i } _{i = 1, ..., N} , where i indicates the sort subscript of the input haptic signal, and N is the total amount of the input haptic signal. After passing through the encoding module E _h (·) of the autoencoder, f _i ^h = E _h (h _i ; θ _he ) is the feature representation of the haptic signal h _i , where f ^h = {f _i ^h } _{i = 1, ..., N} , where θ _he is the parameter of the encoding module. f _i ^h is input to the decoding module D _h (·), and the output haptic signal
where θ _hd is the parameter of the decoding module, and performs clustering based on the K-means algorithm on the features f _i ^h to output the corresponding category tags s _i ^h . The parameters in the above process are jointly estimated, and the loss function
(however,
is the reconstruction error of the encoder, N is the number of haptic signals,
is the clustering error of K-means, M is the cluster center vector matrix for obtaining tactile data using the K-means algorithm, m _c in the c-th column of the M matrix indicates the center of mass of the c-th cluster, θ _h = [θ _he , θ _hd ] is the parameter of the encoder module and the decoder module, s _j,i ^h is the j-th element of s _i ^h , and if the s _j,i ^h value of the element is 1 and the other elements are all 0, it indicates that the original tactile signal h _i corresponding to s _i ^h belongs to the j-th category, and l is the least squares loss.
where λ is a regularization parameter, λ≧0;
3. The method for reconstructing a fine-grained haptic signal for audiovisual assistance according to claim 2, further comprising minimizing L _clu ^h to estimate θ _h and obtain f _i ^h and s _i ^h . Transferring the feature extraction ability of a haptic autoencoder to an audio feature extraction network and an image feature extraction network using a cross-modal transfer learning method is
The feature self-adaptation method minimizes the maximum average difference criterion between the tactile and audiovisual areas to achieve the transition, i.e.,
Let the distributions of the haptic, audio, and image signal sets be P, Q, and R, respectively, denote the MMD between the haptic and audio signals as MMD _k (P,Q), the MMD between the haptic and visual signals as MMD _k (P,R), and in the reproduction kernel Hilbert space H _k , H _k contains a function set f defined on a non-empty set, and the square of the MMD is
and
where haptic, audio, and image signals are passed through respective feature extraction networks φ to obtain extracted feature vectors, the haptic feature vector is denoted as ^φh (h; _θhe ), i.e., it is the output of the encoding module of the autoencoder, the audio and image feature vectors are ^φa (a; _θa )= ^fa , ^φv (v; _θv )= ^fv , the three modal feature set is denoted as ( ^fh , ^fa , ^fv ), _θa and _θv are the parameters of the audio and image feature extraction networks respectively, _θhe is the parameter of the encoding module, and for any function f∈Hk and any _X∈P ,
where μ _k (P) is the mean embedding of P in H _k , i.e., one element representation in H _k -space of the distribution P, f(X) denotes the mapping of X into H _k- space by function f, <·,·> _{H k} is the dot product operation, and similarly,
and μ _k (Q) is the mean embedding of Q in H _k ,
and μ _k (R) is the mean embedding of R in H _k ;
handle
The value of is calculated as the loss function of cross-modal transfer, and the specific formula is
And,
Optimizing the _LCT guides information flow between the haptic feature extraction autoencoder model and the audio-visual feature extraction network, effectively transferring the feature extraction capability of the autoencoder for the haptic modality to the audio-visual feature extraction network, i.e., estimating θ _a and θ _v . Further optimizing the audio feature extraction network and the image feature extraction network includes:
The classification loss function is
(where s _i ^a and s _i ^v are the category tags of the audio signal and the image signal, respectively; by minimizing L _clu ^av , the optimal values of θ _a and θ _v are further obtained; p(f _i ^a ; θ _a ) means the probability of obtaining the audio signal category s _i ^a when the audio feature extraction network input is f _i ^a and the network parameter is θ _a ; p(f _i ^v ; θ _v ) means the probability of obtaining the image signal category s _i ^v when the image feature extraction network input is f _i ^v and the network parameter is θ _v ; f ^a = {f _i ^a } _{i = 1, ... , N} , f ^v = {f _i ^v } _{i = 1, ... , N} );
Combining L _clu ^h , L _{CT ,} and L _clu ^av ;
Obtaining the total objective function L _clu =L _clu ^h +L _CT +L _clu ^av ;
The method for reconstructing fine-grained haptic signals for audiovisual assistance according to claim 4, further comprising: minimizing L _clu to obtain optimal θ _h , θ _{a ,} and θ _v ; and after determining the parameters, obtaining features f ^h , ^fa , and f ^v corresponding to the haptic signal, audio signal, and image signal. By constraining the extracted tactile, audio, and image modal features by jointly considering the clustering constraint, the centrality constraint, and the sorting constraint, each modal feature belonging to the same meaning is brought close to each other, but each modal feature not belonging to the same meaning is separated, and tactile, audio, and image modal features with fine-grained classification are obtained.
To ensure compactness between features of the same fine-grained subcategory, clustering learning under center constraints is performed on three types of modal signals. To achieve better fine-grained classification performance, features of the same subcategory should be adjacent in a common space, with the goal of minimizing intra-category variance. Clustering learning is driven by minimizing the distance from a feature to its subcategory center. The subcategory center of the tactile signal is set as the common subcategory center of the cross-modal signal to ensure compactness between semantic features of the same category of cross-modal signals. The loss function of clustering learning under center constraints is:
is defined as
2. The method for reconstructing fine-grained haptic signals for audiovisual assistance according to claim 1, wherein N is the number of haptic, audio, and image signals, s _i ^h , s _i ^a , and s _i ^v indicate the categories of haptic signals, audio signals, and image signals, respectively, the audio signals and image signals share a cluster center vector matrix M of haptic signals, and the above process includes bringing each modal feature f _i ^a , f _i ^v , and f _i ^h with similar meaning closer to each other. To ensure that the features of different fine-grained subcategories have a certain sparsity, we perform clustering learning under sorting constraints on three types of modal signals. The goal of the centrality constraint is to minimize the intra-category variance, while the goal of the sorting constraint is to maximize the inter-category variance, so that the feature outputs of different subcategories are less similar than the feature outputs of the same subcategory. The sorting constraint is:
is defined as
Here, C is the total number of categories after the haptic signal is clustered using the K-means algorithm, and the above process brings modal features f _i ^a , f _i ^v , and f _i ^h with similar meanings closer together, while separating modal features with different meanings as much as possible. Constructing a triplet set based on tactile, audio, and image modal features, performing shared semantic learning of triplet constraints, optimizing a multimodal fusion mapping function, and obtaining fusion features containing shared semantic information is
Randomly selecting one haptic signal sample h _i from a fine-grained subcategory and setting the sample as an anchor;
Selecting a sample v _i ⁺ from the image dataset that belongs to the same category as h _i and whose semantic features are closest to f _i ^h as a positive match sample, and selecting a sample v _j ⁻ that does not belong to the same category as h _i and whose semantic features are closest to f _i ^h as a negative match sample;
thereby constructing a set of triplets {(h _i ,v _i ⁺ ,v _j ⁻ )} for the samples in the dataset;
Similarly, obtaining a set of triplets {(h _i , a _i ⁺ , a _j ⁻ )} consisting of haptic signal samples and audio signal samples;
Semantic features f _i ^h of anchor point h _i and positive match semantic features in the corresponding subclass in the audio-visual modal
and minimize the distance between f _i ^h and the semantic feature of negative match
By maximizing the semantic feature between and there is one minimum interval δ, the two triplet loss functions obtained are
and
We introduce an integration paradigm to achieve high-level fusion of multimodal features, i.e.,
By fusing f ^a and f ^v , the process becomes
f ^m =F _m (f ^a ,f ^v ;θ _m )
where f ^m is the output of multimodal fusion in the shared semantic subspace, i.e., the fusion feature, F _m (·) is the multimodal fusion mapping function with parameter θ _m , and F _m (·) takes a linear weighting of f ^a and f ^v ;
fusing f ₊ ^a and f ₊ ^v into f ₊ ^m , and f _- ^a and f _- ^v into f _- ^m ;
The triplet loss is used to constrain the fused features, i.e.,
and
The objective function of shared semantic learning is modeled by combining three loss functions.
It can be shown that L _Syn =L _syn ^a +L _syn ^v +L _syn ^m ;
and minimizing L _Syn to obtain the optimal θ _m . Inputting the fused features into a haptic generation network to reconstruct the haptic signal is
Constructing a haptic generation network, i.e., constructing a haptic generation network G(·), whose structure is the same as D _h (·), and whose network parameters θ _hd are set as the initial values of parameters θ _G of G(·);
The fused features containing the required semantic information are input into a haptic generation network G(·) to obtain the desired haptic signal h′, and the generated haptic signal h′ is remapped to a haptic feature f ^h′ by E _h (·), and a category center is selected to perform semantic constraints on the haptic feature f ^h′ , so that the final loss function is
is shown as
however,
and
indicates the similarity between features f ^h and f ^h′ ,
is the clustering loss of f ^h' , and they are used together as regularization terms of a loss function, and by optimizing the loss function, an optimal value of θ _G is obtained, i.e., G(·) is determined. Pre-configure the parameters of the haptic autoencoder, audio feature extraction network, and image feature extraction network.
Presetting the parameters of the multimodal fusion mapping function and the parameters of the haptic generation network;
Training the haptic autoencoder, the audio feature extraction network, the image feature extraction network, the multimodal fusion mapping function, and the haptic generation network includes step 1 and step 2;
In the step 1, θ _v , θ _a , θ _he , θ _hd , and M are preset, and {s _i ^h }, {s _i ^a }, and {s _i ^v } are optimized.
Step 11: Initialize network parameters θ _v , θ _a , θ _he , θ _hd , node system tags {s _i ^h }, { ^{s ia} _} , {s _i ^v }, and category center matrix M, and set the number of clusters C, learning rate μ _1, and number of iterations T;
{s _i ^h }, {s _i ^a }, {s _i ^v }, and M are fixed, and θ _v , θ _a , θ _he , and θ _hd are optimized based on stochastic gradient descent, i.e.,
and
where step 12 is where ∇ is the partial derivative of each loss function;
Fix θ _v , θ _a , θ _he , θ _hd , and M, and optimize {s _i ^h }, {s _i ^a }, {s _i ^v }, i.e.,
Step 13, where
Fix θ _v , θ _a , θ _he , θ _hd , and {s _i ^h }, {s _i ^a }, {s _i ^v } and optimize M, i.e.,
Step 14, where
If t<T, jump to step 412; if t=t+1, continue with the next iteration; otherwise, end the iteration;
and step 16, after T iterations, obtaining optimal audio feature extraction network parameters θ _a , image feature extraction network parameters θ _v , haptic autoencoder parameters θ _he , θ _hd , node system tags {s _i ^h }, {s _{i a} ^} , {s _i ^v }, and a cluster center vector matrix M of the haptic data;
In the step 2, θ _m and θ _G are estimated based on a stochastic gradient descent method, and the step 2 includes the following steps:
Step 21 of initializing θ _m , learning rates μ ₂ , μ ₃ , and number of iterations n ₁ ;
Based on L _Syn, θ _m is estimated using stochastic gradient descent, i.e.,
Step 22, where
Based on L _Gen , θ _G is updated using stochastic gradient descent, i.e.,
Step 23, where
If n< _n1 , jump to step 22; if n=n+1, continue with the next iteration; otherwise, step 24 to end the iteration;
and (25) obtaining optimal θ _m and θ _G after _n iterations.