CN115936997A - An image super-resolution reconstruction method for cross-modal communication - Google Patents

Abstract

The invention belongs to the technical field of super-resolution reconstruction of visual signals, and discloses an image super-resolution reconstruction method for cross-modal communication. First, only low-resolution visual signals and corresponding tactile signals are transmitted at the sending end, and then through the The learning of intra-modal discrimination and inter-modal correlation bridges the semantic gap between different modalities. After transmission through the channel, complementary learning is realized through effective feature fusion at the receiving end, and finally the obtained fusion features are used to generate High-resolution visual signals required. The present invention well solves the problems in the multi-modal service that the visual quality decreases due to the limited bandwidth and the competition between multi-modal signals, which ultimately affects the user experience, and realizes the consistency and complementarity of the cross-modal signals Advanced learning ensures the acquisition of high-resolution visual signals at the receiving end under limited bandwidth, improving the user's immersive experience.

Description

An image super-resolution reconstruction method for cross-modal communication

Technical Field

The present invention belongs to the technical field of super-resolution reconstruction of visual signals, and specifically relates to an image super-resolution reconstruction method for cross-modal communication.

Background Art

With the rapid development of wireless and multimedia communication technologies, human audiovisual needs have been greatly met, and people have begun to pursue more diversified and richer experiences. When tactile signals are combined with traditional audiovisual signals, multimodal services emerge, which can provide more fine-grained interactions and immersive experiences. Many studies have found that in online multimodal service scenarios, people can improve their perception and interactive experience of products through high-resolution visual signals and high-fidelity tactile signals. For example, in online shopping, consumers can obtain detailed information about product details, intrinsic perceived texture, hardness and other characteristics through touch and observation. In order to support multimodal services, cross-modal communication has emerged, which ensures the coordinated transmission and processing between multiple modalities by exploiting the correlation between different modalities. However, the limited bandwidth and the existence of inter-modal competition make it difficult to implement existing cross-modal communication schemes, which will lead to a decline in the user's immersive experience, especially an unsatisfactory visual experience.

Specifically, on the one hand, high-fidelity visual signals are an important guarantee for user immersive experience. However, due to limited bandwidth, it is difficult to transmit such high-resolution pictures/videos in online multimedia communication services. On the other hand, there is competition between visual and tactile modalities during the transmission process. In order to meet the requirements of low latency and high reliability of tactile signals, existing solutions usually give tactile signals a higher priority. However, the frequent and irregular appearance of tactile signals will seriously affect the transmission quality of visual signals, especially when users have frequent touch needs, such as online shopping.

At present, the problem of visual signal quality degradation caused by insufficient bandwidth and competition between modalities can be solved from two perspectives: multimodal communication and super-resolution reconstruction. Multimodal communication solutions mainly include traditional audio and video communication solutions and tactile communication solutions. These solutions can achieve high-fidelity transmission of audio, video or tactile signals alone, but when it comes to scenarios where audio, video and tactile signals are transmitted simultaneously, the quality of the receiving end cannot be guaranteed; super-resolution reconstruction solutions mainly use low-resolution vision to complete the reconstruction of high-resolution visual signals based on a single visual signal or based on reference information (such as visual signals at different angles, adjacent frames, boundary maps, etc.), but most of them complete the reconstruction task at the local terminal and do not involve communication tasks.

The above-mentioned existing multimodal transmission schemes under limited bandwidth have the following main defects: each mode is considered separately, and the consistency and complementarity between multimodal data are not reasonably utilized; the communication process is not considered, and the data is only processed at the terminal, and the competition between modes in the multimodal data transmission process is not considered.

Summary of the invention

In order to overcome the shortcomings of the prior art, the present invention provides an image super-resolution reconstruction method for cross-modal communication, which relies on the semantic consistency between tactile signals and visual signals, and realizes the extraction and mapping of each modal feature by fully considering the relationship within and between modalities. Then, with the help of a powerful feature fusion network, it is effectively realized to use low-resolution visual signals and tactile signals to generate fusion features that are as similar as possible to high-resolution visual signal features, and finally obtain high-resolution visual signals, thereby ensuring the user's immersive experience in bandwidth-limited multimodal application scenarios.

In order to achieve the above object, the present invention is achieved through the following technical solutions:

The present invention is an image super-resolution reconstruction method for cross-modal communication, comprising the following steps:

Step (1), using the complete high-resolution visual signal, encoding and decoding the high-resolution visual signal, training the encoding network of the high-resolution visual signal through the encoding step, and obtaining the encoding features of the high-resolution visual signal, and training the decoding (generation) network of the high-resolution visual signal through the decoding step, so as to provide support for the subsequent visual signal super-resolution reconstruction model;

Step (2), design a haptic-acid super-resolution reconstruction HaSR (Haptic-acid Super-resolution Reconstruction) model for visual signals; the super-resolution reconstruction HaSR model is as follows:

After the visual signal and the tactile signal are collected from the terminal, the visual signal is first downsampled at the edge node to obtain a low-resolution visual signal; further, a pre-trained and widely used encoding network is used to perform preliminary feature extraction on the low-resolution visual signal and the corresponding tactile signal;

Then, by using the intra-modality discriminability and inter-modality consistency, the mapping network is used to reduce the difference between modalities, and the correlation between different modalities is mined to fill the semantic gap between modalities. Based on the preliminary features extracted by encoding, mapping features with semantic discrimination and semantic association are obtained. The obtained mapping features are then normalized and passed through the channel model for the next step of feature fusion.

Based on the mapping features of low-resolution visual signals and tactile signals and the encoding features of high-resolution visual signals, combined with the powerful data fitting ability of the generative adversarial network, the fusion features are obtained;

Finally, the obtained fusion features are input into the generation network of high-resolution visual signals to achieve reconstruction of high-resolution visual signals;

Step (3), using the optimization method to train the HaSR model, and finally obtaining the optimal model parameters for the subsequent testing phase;

Step (4): input the paired low-resolution visual signal and tactile signal to be tested into the optimal HaSR model. The optimal HaSR model is used to extract the features of the low-resolution visual signal and the corresponding tactile signal and fuse them, and use the fused features to generate the required high-resolution visual signal.

A further improvement of the present invention is that step (1) comprises the following steps:

其包含配对的触觉、低分辨视觉信号和高分辨率视觉信号，N为配对的视觉信号和触觉信号的数量，d_i＝{h_i,l_i,t_i}分别代表高分辨率视觉信号、低分辨率视觉信号和对应的触觉信号，将第i个高分辨率视觉信号h_i输入高分辨率视觉信号的编码网络，提取视觉信号的编码特征z_h；(1-1) For the training data set

It includes paired tactile, low-resolution visual signals and high-resolution visual signals, N is the number of paired visual signals and tactile signals, d _i ={h _i ,l _i ,t _i } respectively represents the high-resolution visual signal, the low-resolution visual signal and the corresponding tactile signal, the i-th high-resolution visual signal h _i is input into the encoding network of the high-resolution visual signal, and the encoding feature z _h of the visual signal is extracted;

(1-2), the obtained encoding feature z _h of the high-resolution visual signal is input into the decoder of the high-resolution visual signal composed of the generative adversarial network, and then the high-resolution visual signal reconstructed by the decoder is input into the discriminator of the high-resolution visual signal. By jointly training the encoder and decoder, and optimizing with reconstruction loss and identification loss, the encoding feature of the high-resolution visual signal and the corresponding decoding network are finally learned; specifically, the loss function is defined as follows:

L _pre = L _rec + α L _pre-adv ,

Where α is a coefficient used to adjust the proportion of different losses. The first loss is the reconstruction loss:

代表相应的编码网络得到的高分辨率视觉信号的编码特征，||·||₁代表相应的L₁范数；第二项是生成对抗网络的损失,具体的损失函数定义如下：Where (C, H, W) is the size of the high-resolution visual signal, _Gh represents the corresponding decoding (generation) network of the high-resolution visual signal,

represents the encoding features of the high-resolution visual signal obtained by the corresponding encoding network, and ||·|| ₁ represents the corresponding _L1 norm; the second term is the loss of the generative adversarial network, and the specific loss function is defined as follows:

Where E(*) represents the expected value of the distribution function, p(z _h ) represents the distribution of high-resolution image features, p(h) represents the distribution of real high-resolution images, D _h represents the corresponding high-resolution visual signal identification network, which is used to complete the judgment of the reconstructed high-resolution visual signal, D _h represents the corresponding high-resolution visual signal identification network, which is used to complete the judgment of the reconstructed high-resolution visual signal, θ _gh and θ _dh represent the parameters of the corresponding high-resolution visual signal generator and discriminator, respectively. By minimizing L _pre, the optimal high-resolution visual signal encoding network and the corresponding high-resolution visual signal encoding features, the corresponding decoding network (i.e., the generation network) and the corresponding high-resolution visual signal identification network are obtained.

A further improvement of the present invention is that step (2) comprises the following steps:

中存在的配对的低分辨率视觉信号l_i和触觉信号h_i，利用现有的成熟的深度神经网络，完成低分辨率视觉信号和触觉信号初步的特征提取,获得相应的低分辨率视觉信号的编码特征f_l和对应的触觉信号编码特征f_t；(2-1) Preliminary feature extraction of low-resolution visual signals and corresponding tactile signals, based on

The paired low-resolution visual signal l _i and tactile signal h _i in the image are used to perform preliminary feature extraction of the low-resolution visual signal and the tactile signal using the existing mature deep neural network, and obtain the corresponding encoding feature f _l of the low-resolution visual signal and the corresponding encoding feature f _t of the tactile signal;

(2-2) Based on the obtained low-resolution visual signal encoding feature _fl and the corresponding tactile signal encoding feature _ft , a feature mapping network is established to effectively reduce the heterogeneity between modalities, learn the intra-modal identification representation and the inter-modal consistent representation from the perspectives of the same modality and cross-modality, and finally obtain the mapping features, which include the mapping features _zl of the low-resolution visual signal and the mapping features _zt of the tactile signal. Then, the obtained mapping features are input into the channel model, and the corresponding feature fusion step is performed after the main terminal receives them;

Cross-modal semantic relevance learning: Triplet loss is used for cross-modal semantic relevance learning. After learning the mapping network, the following effect is achieved: for low-resolution visual signal features and tactile signal features from the same category, the distance between them should be close, and for low-resolution visual signal features and tactile signal features from different categories, the distance between them should be far away; the following loss function is defined:

Where θ _l and θ _t represent the parameters of the corresponding low-resolution visual signal mapping network and tactile signal mapping network, respectively, p and q represent different categories, N represents the number of instances of the corresponding low-resolution visual signal and tactile signal, σ represents the corresponding threshold, L ₂ =||·|| ₂ represents the corresponding L ₂ norm, and the total loss of semantic relevance can be expressed as the sum of the above two, that is:

Discriminative learning within the same modality: While ensuring semantic relevance, the semantic discrimination problem within the same modality should be effectively solved. That is, for samples within the same modality (visual modality or tactile modality), samples belonging to the same category should be closer, and samples belonging to different categories should be farther away. This is mainly achieved by adding a common classifier after the mapping network. The specific loss is expressed as follows:

Wherein, p _i (z) represents the probability distribution predicted by the classifier, _yi is the real label, and θ _c represents the parameter of the corresponding public classifier; after the above processing, the mapping features z _l of the low-resolution visual signal and the mapping features z _t of the tactile signal are normalized and then input into the channel model;

(2-3) After the transmission through the channel, the corresponding mapping features z _ln of the noisy low-resolution visual signal and z _tn of the tactile signal are obtained at the main terminal. The purpose of this step is to use the complementarity of the two to complete the generation of fusion features, so that the generated fusion features are as similar as possible to the encoding features z _h of the high-resolution visual signal in step (1). Based on this purpose, the ability of the generative adversarial network in fitting data distribution is used to complete the feature fusion task; where z _h is regarded as a real sample, z _ln and z _tn are regarded as the input of the generator, z _m represents the obtained fusion feature, and the loss of the defined fusion network is as follows:

L _m =L _m-adv (G _m ,D _m )+L ₂ (z _m ,z _h ),

The first term represents the loss of the ordinary generative adversarial network, _Gm represents the feature fusion network, and _Dm represents the feature identification network corresponding to the feature fusion network. The specific expression of this term is:

Where θ _gm represents the corresponding parameters of the fusion network generator, and θ _dm represents the model parameters of the identification network corresponding to the fusion network generator; the second term represents the L ₂ loss, which is conducive to maintaining semantic consistency;

(2-4) After obtaining the fusion features, the fusion features are used to reconstruct the high-resolution visual signal. The high-resolution visual signal decoding (generation) network obtained in the first step and the corresponding identification network are used, and their network structure and parameters are used as the initialization network parameters of this step. In addition, based on the previous loss, the perceptual loss is added to make the generated visual signal more consistent with human perceptual characteristics. The loss of this step is specifically expressed as follows:

L _finet =L _per +βL _adv-finet +γL _rec ,

The first term is the perceptual loss, which can be specifically expressed as:

Where Mi _,j represents the number of parameters of the corresponding feature map, Fi ^,j represents the output of the VGG-19 network after the i-th convolutional layer and before the j-th maximum pooling layer, and the second term is the loss of the generative network of the high-resolution visual signal, which can be specifically expressed as

Where θ _gh represents the parameters of the corresponding generative network that uses the fused features z _m to reconstruct high-resolution visual signals, θ _dh represents the parameters of the corresponding discriminative network, and the third term is the reconstruction loss; in addition, β and γ are hyperparameters.

A further improvement of the present invention is that step (3) comprises the following steps:

(3-1) Using the existing high-resolution visual signals, train the encoding network of the high-resolution visual signals and the corresponding decoding (generation) network, as well as the corresponding identification network of the high-resolution visual signals. The specific process is as follows:

Step 311, initializing parameters _θeh (0), _θgh (0), _θdh (0) to the values of the corresponding parameters at the 0th iteration;

Step 312, setting the number of iterations to n ₁ and the learning rate to μ ₁ ;

Step 313: Use RMSProp algorithm to optimize network parameters:

为对各个损失函数做偏导；Wherein _θeh (n+1), _θdh (n+1), _θgh (n+1) and _θeh (n), _θdh (n), _θgh (n) are the parameters of the encoding network of the high-resolution visual signal, the identification network of the high-resolution visual signal, and the generation network of the high-resolution visual signal corresponding to the n+1th and nth times, respectively.

To make partial derivatives for each loss function;

Step 314: if n<n ₁ , repeat step 313. After n ₁ rounds of iteration, a network that converges to the best is obtained, including a coding network for high-resolution visual signals, a discrimination network for high-resolution visual signals, and a generation network for high-resolution visual signals.

(3-2) Based on the encoding network of the high-resolution visual signal obtained in the first step, the decoding (generation) network of the high-resolution visual signal and the corresponding identification network, the low-resolution visual signal and the corresponding tactile signal are used to train the corresponding low-resolution visual signal encoding and mapping network, the tactile signal encoding and mapping network, and the feature fusion network, and fine-tune the corresponding high-resolution visual signal generation network, wherein the low-resolution visual signal encoding network and the tactile signal encoding network are obtained by loading the pre-trained network and do not participate in the update in this step; the specific process is as follows:

Step 321, initializing parameters θ _l (0), θ _t (0), θ _c (0), which represent the initial random parameters of the corresponding network, and loading the parameters of the generation network and the identification network of the high-resolution visual signal obtained in step 3-1 as the initial parameters of θ _gh (0), θ _dh (0);

Step 322, start iteration, set the number of iterations to n ₂ and the learning rate to μ ₂ ;

Step 323: Adopt the Adam algorithm to optimize the parameters of the low-resolution visual signal mapping network, the tactile signal mapping network, and the common classifier:

为对各个损失函数做偏导；where θ _c (n+1), θ _l (n+1), θ _t (n+1) and θ _c (n), θ _l (n), θ _t (n) are the parameters of the n+1th and nth corresponding public classifiers, the low-resolution visual signal mapping network and the corresponding tactile signal mapping network, respectively.

To make partial derivatives for each loss function;

Step 324: Adopt the Adam algorithm to optimize the parameters of the feature fusion network and the corresponding fusion feature identification network:

为对各个损失函数做偏导；Among them, θ _gm (n+1), θ _dm (n+1), θ _gm (n) and θ _dm (n) are the parameters of the generative network and the discriminative network of the fusion features corresponding to the n+1th and nth times, respectively.

To make partial derivatives for each loss function;

Step 325: Use the RMSProp algorithm to fine-tune the generation network and the identification network of the high-resolution visual signal, and optimize and update them through the following function:

为对各个损失函数做偏导；where _θgh (n+1), _θdh (n+1), _θgh (n) and _θdh (n) represent the parameters of the generation network and the discrimination network corresponding to the n+1th and nth high-resolution visual signals, respectively.

To make partial derivatives for each loss function;

Step 326: If n<n ₂ , jump to step 323. After n ₂ rounds of iteration, a HaSR network that converges to the optimal is obtained, including a mapping network for low-resolution visual signals, a mapping network for tactile signals, a feature fusion network and a corresponding identification network for fused features, and an identification network for high-resolution visual signals that converges to the optimal after fine-tuning, and a generation network for high-resolution visual signals.

A further improvement of the present invention is that step (4) comprises the following steps:

(4-1) Using the trained HaSR model;

(4-2) A set of paired low-resolution visual signals and corresponding tactile signals are input into the HaSR model to complete the encoding, mapping, and fusion of modal features, and finally obtain the corresponding high-resolution visual signals.

The beneficial effects of the present invention are:

The present invention can overcome the ill-posed problem caused by the traditional single-modal super-resolution reconstruction of visual signals by matching tactile signals with visual signals, that is, a low-resolution visual signal may correspond to multiple high-resolution visual signals;

The present invention overcomes the heterogeneity of different modalities by fully exploring the relationship between visual signals and tactile signals within and between modalities;

The present invention relies on an effective feature fusion method and can fully utilize the complementarity of different modalities to improve the quality of the generated high-resolution visual signal.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG1 is a flow chart of an image super-resolution reconstruction method according to the present invention.

FIG. 2 is a schematic diagram of a complete network structure of the present invention.

FIG. 3 is a diagram showing super-resolution reconstruction results of the present invention and other comparative methods.

DETAILED DESCRIPTION

In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and specific implementation examples.

The present invention provides an image super-resolution reconstruction method for cross-modal communication, the flow chart of which is shown in FIG1 . The method comprises the following steps:

Step 1: Use the complete high-resolution visual signal to encode and decode the high-resolution visual signal, that is, stage 1 in Figure 2. Through the encoding step, the encoding network of the high-resolution visual signal is trained to obtain the encoding features of the high-resolution visual signal. Through the decoding step, the decoding (generation) network of the high-resolution visual signal is trained to provide support for the subsequent visual signal super-resolution reconstruction model.

其包含配对的触觉、低分辨视觉信号和高分辨率视觉信号，N为配对的视觉信号和触觉信号的数量，d_i＝{h_i,l_i,_ti}分别代表高分辨率视觉信号128×128、低分辨率视觉信号32×32和对应的触觉信号，通过短时傅里叶变换得到的频谱图，将第i个高分辨率视觉信号h_i输入高分辨率视觉信号的编码网络，编码网络由在ImageNet上预训练的VGG-16构成，其后添加维度分别为512和128的全连接层，提取视觉信号的编码特征z_h，其维度为128；Step 1-1: For the training data set

It contains paired tactile, low-resolution visual signals and high-resolution visual signals, N is the number of paired visual signals and tactile signals, d _i ={h _i ,l _i , _t i} respectively represents the high-resolution visual signal 128×128, the low-resolution visual signal 32×32 and the corresponding tactile signal, and the spectrum obtained by short-time Fourier transform is used to input the i-th high-resolution visual signal h _i into the encoding network of the high-resolution visual signal. The encoding network is composed of VGG-16 pre-trained on ImageNet, and then fully connected layers with dimensions of 512 and 128 are added to extract the encoding feature z _h of the visual signal, whose dimension is 128;

Step 1-2, input the obtained encoding feature z _h of the high-resolution visual signal into the decoder of the high-resolution visual signal composed of the generative adversarial network, which is composed of an inverse convolutional network with a 3×3 convolution kernel, and the corresponding filter sizes are 512, 256, 128, 64, and 3 respectively. Then input the high-resolution visual signal reconstructed by the decoder into the discriminator of the high-resolution visual signal. The discriminator network is stacked by convolutional networks, and its convolution kernel size is 3×3. The corresponding filter data volume is 64, 128, 256, and 512. Then, a fully connected layer with dimensions of 512, 128, and 1 is added. By jointly training the encoder and decoder, and optimizing with reconstruction loss and identification loss, the encoding features of the high-resolution visual signal and the corresponding decoding network are finally learned; specifically, the defined loss function is as follows:

L _pre = L _rec + α L _pre-adv ,

Where α is a coefficient used to adjust the proportion of different losses. The first loss is the reconstruction loss:

代表相应的编码网络得到的高分辨率视觉信号的编码特征，||·||₁代表相应的L₁范数；第二项是生成对抗网络的损失,具体的损失函数定义如下：Where (C, H, W) is the size of the high-resolution visual signal, _Gh represents the corresponding decoding (generation) network of the high-resolution visual signal,

represents the encoding features of the high-resolution visual signal obtained by the corresponding encoding network, and ||·|| ₁ represents the corresponding _L1 norm; the second term is the loss of the generated adversarial network, and the specific loss function is defined as follows:

Where E(*) represents the expected value of the distribution function, p(z _h ) represents the distribution of high-resolution image features, p(h) represents the distribution of real high-resolution images, D _h represents the corresponding high-resolution visual signal identification network, which is used to complete the judgment of the reconstructed high-resolution visual signal, D _h represents the corresponding high-resolution visual signal identification network, which is used to complete the judgment of the reconstructed high-resolution visual signal, θ _gh and θ _dh represent the parameters of the corresponding high-resolution visual signal generator and discriminator, respectively. By minimizing L _pre, the optimal high-resolution visual signal encoding network and the corresponding high-resolution visual signal encoding features, the corresponding decoding network (i.e., the generation network) and the corresponding high-resolution visual signal identification network are obtained.

Step 2: Design a tactile-assisted visual signal super-resolution reconstruction HaSR model; the structure diagram of the super-resolution reconstruction HaSR model is shown in stage 2 of Figure 2:

After the visual signal and the tactile signal are collected from the terminal, the visual signal is first downsampled at the edge node to obtain a low-resolution visual signal; further, a pre-trained and widely used encoding network is used to perform preliminary feature extraction on the low-resolution visual signal and the corresponding tactile signal;

Then, by using the intra-modality discriminability and inter-modality consistency, the mapping network is used to reduce the difference between modalities, and the correlation between different modalities is mined to fill the semantic gap between modalities. Based on the preliminary features extracted by encoding, mapping features with semantic discrimination and semantic association are obtained. The obtained mapping features are then normalized and passed through the channel model for the next step of feature fusion.

Based on the mapping features of low-resolution visual signals and tactile signals and the encoding features of high-resolution visual signals, combined with the powerful data fitting ability of the generative adversarial network, the fusion features are obtained;

Finally, the obtained fusion features are input into the generative network of high-resolution visual signals to achieve reconstruction of high-resolution visual signals.

The specific implementation steps are as follows:

中存在的配对的低分辨率视觉信号l_i和触觉信号h_i，利用现有的成熟的深度神经网络具体来说对触觉信号来说使用DenseNet-121网络对于低分辨率视觉信号采用VGG-16网络，完成低分辨率视觉信号和触觉信号初步的特征提取,获得相应的低分辨率视觉信号的512维编码特征f_l和对应的512维触觉信号编码特征f_t；Step 2-1: Preliminary feature extraction of low-resolution visual signals and corresponding tactile signals, based on

The paired low-resolution visual signal l _i and tactile signal h _i in the image are used to extract the initial features of the low-resolution visual signal and the tactile signal by using the existing mature deep neural network. Specifically, the DenseNet-121 network is used for the tactile signal and the VGG-16 network is used for the low-resolution visual signal. The corresponding 512-dimensional encoding feature f _l of the low-resolution visual signal and the corresponding 512-dimensional encoding feature f _t of the tactile signal are obtained.

Step 2-2: Based on the obtained low-resolution visual signal coding feature _fl and the corresponding tactile signal coding feature _ft , a feature mapping network is established, which includes a low-resolution visual signal feature mapping network composed of a 512-256-128-dimensional fully connected layer and a tactile signal mapping network composed of a 512-256-128-dimensional fully connected layer, so as to effectively reduce the heterogeneity difference between modalities, learn the identification representation within the modality and the consistent representation between the modalities from the perspectives of the same modality and cross-modality, and finally obtain mapping features, which include a 128-dimensional low-resolution visual signal mapping feature _zl and a 128-dimensional tactile signal mapping feature _zt , and then input the obtained mapping features into the channel model, and perform the corresponding feature fusion step after the master terminal receives them;

Cross-modal semantic relevance learning: Triplet loss is used for cross-modal semantic relevance learning. After learning the mapping network, the following effect is achieved: for low-resolution visual signal features and tactile signal features from the same category, the distance between them should be close, and for low-resolution visual signal features and tactile signal features from different categories, the distance between them should be far away; the following loss function is defined:

Where θ _l and θ _t represent the parameters of the corresponding low-resolution visual signal mapping network and tactile signal mapping network, respectively, p and q represent different categories, N represents the number of instances of the corresponding low-resolution visual signal and tactile signal, σ represents the corresponding threshold, L ₂ =||·|| ₂ represents the corresponding L ₂ norm, and the total loss of semantic relevance can be expressed as the sum of the above two, that is:

Discriminative learning within the same modality: While ensuring semantic relevance, the semantic discrimination problem within the same modality should be effectively solved, that is, for samples within the same modality (visual modality or tactile modality), samples belonging to the same category should be closer, and samples belonging to different categories should be farther away. This is mainly accomplished by adding a common classifier after the mapping network. The classifier consists of 128, 32, and 9-dimensional fully connected layers. The specific loss is expressed as follows:

Wherein, p _i (z) represents the probability distribution predicted by the classifier, _yi is the real label, and θ _c represents the parameter of the corresponding public classifier; after the above processing, the mapping features z _l of the low-resolution visual signal and the mapping features z _t of the tactile signal are normalized and then input into the channel model;

Step 2-3, after the transmission through the channel, the corresponding mapping features z _ln of the noisy low-resolution visual signal and z _tn of the tactile signal are obtained at the main terminal. The purpose of this step is to use the complementarity of the two to complete the generation of fusion features, so that the generated fusion features are as similar as possible to the encoding features z _h of the high-resolution visual signal in step 1. For this purpose, the generative adversarial network is used to fit the data distribution and is selected to complete the feature fusion task; specifically, z _ln and z _tn are first connected in series, and then input into the generation network composed of 256-128 dimensional fully connected layers, where z _h is regarded as a real sample, z _ln and z _tn are regarded as the input of the generator, z _m represents the obtained fusion features, and the loss of the defined fusion network is as follows:

L _m =L _m-adv (G _m ,D _m )+L ₂ (z _m ,z _h ),

The first term represents the loss of the ordinary generative adversarial network, _Gm represents the feature fusion network, and _Dm represents the feature identification network corresponding to the feature fusion network. The network consists of 128, 64, and 1-dimensional fully connected layers. The specific expression of this term is:

Where _θgm represents the corresponding parameters of the fusion network generator, _θdm represents the model parameters of the identification network corresponding to the fusion network generator; the second term represents the _L2 loss, which is conducive to maintaining semantic consistency.

Step 2-4: After obtaining the fusion features, the fusion features are used to reconstruct the high-resolution visual signal. The high-resolution visual signal decoding (generation) network obtained in the first step and the corresponding identification network are used, and their network structure and parameters are used as the initialization network parameters of this step. In addition, based on the loss L _pre in step 1-2, a perceptual loss is added to make the generated visual signal more consistent with human perceptual characteristics. The loss of this step is specifically expressed as follows:

L _finet =L _per +βL _adv-finet +γL _rec ,

The first term is the perceptual loss, which can be specifically expressed as:

Where Mi _,j represents the number of parameters of the corresponding feature map, and F ^i,j represents the output of the VGG-19 network after the i-th convolutional layer and before the j-th maximum pooling layer. Here we set i = 4, j = 5. The second term is the loss of the generative network of the high-resolution visual signal, which can be specifically expressed as

Where θ _gh represents the parameters of the corresponding generative network that uses the fused features z _m to reconstruct high-resolution visual signals, θ _dh represents the parameters of the corresponding discriminative network, and the third term is the reconstruction loss; in addition, β and γ are hyperparameters.

Step 3: Use the optimization method to train the HaSR model and finally obtain the optimal model parameters for the subsequent testing phase;

Step 3-1: Using the existing high-resolution visual signal, train the encoding network of the high-resolution visual signal and the corresponding decoding (generation) network, as well as the corresponding identification network of the high-resolution visual signal. The specific process is as follows:

Step 311, initializing parameters _θeh (0), _θgh (0), _θdh (0) to the values of the corresponding parameters at the 0th iteration;

Step 312, setting the number of iterations to n ₁ =3000 and the learning rate to μ ₁ =0.0008;

Step 313: Use RMSProp algorithm to optimize network parameters:

为对各个损失函数做偏导；Wherein _θeh (n+1), _θdh (n+1), _θgh (n+1) and _θeh (n), _θdh (n), _θgh (n) are the parameters of the encoding network of the high-resolution visual signal, the identification network of the high-resolution visual signal, and the generation network of the high-resolution visual signal corresponding to the n+1th and nth times, respectively.

To make partial derivatives for each loss function;

Step 314: if n<n ₁ , repeat step 313. After n ₁ rounds of iteration, a network that converges to the best is obtained, including a coding network for high-resolution visual signals, a discrimination network for high-resolution visual signals, and a generation network for high-resolution visual signals.

Step 3-2: Based on the encoding network of high-resolution visual signals obtained in the first step, the decoding (generation) network of high-resolution visual signals and the corresponding identification network, the low-resolution visual signals and the corresponding tactile signals are used to train the corresponding low-resolution visual signal encoding and mapping network, the tactile signal encoding and mapping network, and the feature fusion network, and fine-tune the corresponding high-resolution visual signal generation network, wherein the low-resolution visual signal encoding network and the tactile signal encoding network are obtained by loading the pre-trained network and do not participate in the update in this step. The specific process is as follows:

Step 321, initializing parameters θ _l (0), θ _t (0), θ _c (0), which represent the initial random parameters of the corresponding network, and loading the parameters of the generation network and the identification network of the high-resolution visual signal obtained in step 3-1 as the initial parameters of θ _gh (0), θ _dh (0);

Step 322, start iteration, set the number of iterations to n ₂ =2000, and the learning rate to μ ₂ =0.0015;

Step 323: Adopt the Adam algorithm to optimize the parameters of the low-resolution visual signal mapping network, the tactile signal mapping network, and the common classifier:

为对各个损失函数做偏导；where θ _c (n+1), θ _l (n+1), θ _t (n+1) and θ _c (n), θ _l (n), θ _t (n) are the parameters of the n+1th and nth corresponding public classifiers, the low-resolution visual signal mapping network and the corresponding tactile signal mapping network, respectively.

To make partial derivatives for each loss function;

Step 324: Adopt the Adam algorithm to optimize the parameters of the feature fusion network and the corresponding fusion feature identification network:

为对各个损失函数做偏导；Among them, θ _gm (n+1), θ _dm (n+1), θ _gm (n) and θ _dm (n) are the parameters of the generative network and the discriminative network of the fusion features corresponding to the n+1th and nth times, respectively.

To make partial derivatives for each loss function;

Step 325: Use the RMSProp algorithm to fine-tune the generation network and the identification network of the high-resolution visual signal, and optimize and update them through the following function:

为对各个损失函数做偏导；where _θgh (n+1), _θdh (n+1), _θgh (n) and _θdh (n) represent the parameters of the generation network and the discrimination network corresponding to the n+1th and nth high-resolution visual signals, respectively.

To make partial derivatives for each loss function;

Step 326: If n<n ₂ , jump to step 323. After n ₂ rounds of iteration, a HaSR network that converges to the optimal is obtained, including a mapping network for low-resolution visual signals, a mapping network for tactile signals, a feature fusion network and a corresponding identification network for fused features, and an identification network for high-resolution visual signals that converges to the optimal after fine-tuning, and a generation network for high-resolution visual signals.

Step 4: Input the paired low-resolution visual signals and tactile signals to be tested into the optimal HaSR model. The optimal HaSR model is used to extract the features of the low-resolution visual signals and the corresponding tactile signals and fuse them, and use the fused features to generate the required high-resolution visual signals.

Step 4-1: Use the trained HaSR model;

Step 4-2: Input a set of paired low-resolution visual signals and corresponding tactile signals into the HaSR model to complete the encoding, mapping, and fusion of modal features, and finally obtain the corresponding high-resolution visual signals.

The following experimental results show that compared with existing methods, the present invention utilizes the consistency and complementarity of multimodal signals to achieve super-resolution reconstruction of visual signals and achieves better generation effects.

The present invention uses the LMT-108 multimodal dataset to conduct experiments, which is commonly used in cross-modal retrieval and cross-modal generation tasks, and contains 108 materials commonly found in daily life. These different surface materials can be roughly divided into nine categories according to their physical properties, namely, grid, stone, metal, wood, rubber, fiber, foam, foil and paper, textiles and fabrics. Each category further contains 5-17 subcategories. For each category, the dataset includes visual signals of various types of textures, and three-axis (X, Y, Z) acceleration signals generated by sliding or tapping. Based on previous work on cross-modal learning using this dataset, the visual signal samples in the present invention are mainly represented by non-flashing RGB visual signals; for tactile data, the z-axis acceleration signal shows the most obvious vibration during the movement of the collector, so the z-axis acceleration signal is selected as the tactile signal. The original high-resolution visual signal is represented as 128×128 in size, and the low-resolution visual signal is obtained by 4× downsampling, so its size is 32×32. For the tactile signal, considering the pressure effect when the collector touches and leaves, only the middle tactile signal is intercepted, and then we obtain the corresponding spectrum through STFT transformation.

Existing method 1: The PIX2PIX method in the document “Image-to-image translation with conditional adversarial networks” (authors P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros) is a classic method for applying generative adversarial networks (GAN) to supervised conversion of visual signals of different styles. The present invention uses the spectrogram of tactile signals and the paired high-resolution visual signals as paired training data, and uses the spectrogram of tactile signals as conditional information to generate high-resolution visual signals.

Existing method 2: Discogan in the document “Learning to discover cross-domain relations with generative adversarial networks” (authors T. Kim, M. Cha, H. Kim, J. K. Lee, and J. Kim) uses GAN to discover relations between different domains and achieve conversion from one domain to another. The present invention achieves conversion from the spectrogram of tactile signals to high-resolution visual signals.

Existing method 3: Bilinear interpolation is a commonly used visual signal upsampling method, which directly interpolates low-resolution visual signals to obtain clearer visual signals. It has a smoothing function and can effectively overcome the shortcomings of traditional neighborhood interpolation.

Existing method 4: The paper "Photo-realistic single image super-resolution using agenerative adversarial network" (authors C. Ledig, L. Theis and F. Huszar et al.) proposed SRGAN, which is the first method to introduce GAN into the field of super-resolution reconstruction, and obtains visual signals that are more in line with human perception by introducing perceptual loss.

Existing method 5: The document "Esrgan: Enhanced super-resolution generative adversarial networks" (authors X. Wang, K. Yu, S. Wu et al.) introduces a densely connected residual block based on SRGAN to achieve deeper network training and further improve the visual quality by balancing perceptual quality and fidelity.

The present invention: the method of this embodiment.

In this embodiment, the performance indicators used to evaluate the super-resolution reconstruction solution proposed in the present invention are divided into three categories: peak signal-to-noise ratio, structural similarity, and Frechet Inception distance.

Peak signal-to-noise ratio: Peak signal-to-noise ratio (PSNR) is a visual signal quality evaluation indicator. It is based on the error between pixels and calculates the ratio of the peak signal energy to the average energy of the noise. It is the most common and widely used objective evaluation indicator of visual signals. The higher the PSNR value, the smaller the distortion.

Structural similarity: Structural similarity (SSIM) measures the similarity of visual signals from three aspects: brightness, contrast, and structure. The value range of SSIM is [0,1]. The larger the value, the smaller the distortion of the visual signal. Structural similarity is more consistent with the human eye's perception judgment than PSNR in evaluating the quality of visual signals.

Frechet Inception distance: Frechet Inception distance (FID) is used to evaluate the similarity between the visual signal generated by the generative adversarial network and the real visual signal. It calculates the distance between the real visual signal and the generated visual signal in the feature space. First, the Inception network is used to extract features, and then the Gaussian model is used to model the feature space, and then the distance between the two features is solved. Lower FID means higher image quality and diversity.

Table 1 Performance comparison results of super-resolution reconstruction schemes

It can be seen from Table 1 and Figure 3 that compared with the above competitive methods, the method of the present invention has obvious advantages. Among the five comparison schemes, the super-resolution reconstruction method based on cross-modal fusion shows better performance. The results show that the intra-modal and inter-modal mapping feature learning proposed in the present invention can better explore the properties of the modality, and the effective feature fusion method can utilize the complementarity between different modalities. Finally, the reconstruction network is used to improve the reconstruction quality of the visual signal, making it more similar to the original visual signal.

The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered by the protection scope of the present invention.

Claims (7)

1. A cross-modal communication-oriented image super-resolution reconstruction method is characterized by comprising the following steps: the image super-resolution reconstruction method comprises the following steps:

step 1, coding and decoding a high-resolution visual signal by using a complete high-resolution visual signal, training a coding network of the high-resolution visual signal through the coding step, obtaining coding characteristics of the high-resolution visual signal, training a decoding network of the high-resolution visual signal through the decoding step, and providing support for a subsequent visual signal super-resolution reconstruction model;

step 2, designing a touch-assisted visual signal super-resolution reconstruction HasR model; the super-resolution reconstruction HaSR model is as follows:

after the visual signals and the tactile signals are collected from the terminal, the visual signals are down-sampled at the edge nodes to obtain low-resolution visual signals, and the low-resolution visual signals and the corresponding tactile signals are subjected to preliminary feature extraction by utilizing a pre-trained and widely-used coding network;

by utilizing the identification in the modes and the consistency between the modes, the difference between the modes is reduced through a mapping network, and the correlation between different modes is mined to make up the semantic gap between the modes, so that the mapping characteristics with semantic identification and semantic correlation are obtained based on the primary characteristics extracted by coding, and then the obtained mapping characteristics are normalized and pass through a channel model so as to be used for the next step of characteristic fusion;

according to the mapping characteristics of the low-resolution visual signals, the mapping characteristics of the tactile signals and the obtained coding characteristics of the high-resolution visual signals, strong data fitting capacity of the anti-network is generated in a combined mode, and fusion characteristics are obtained;

finally, inputting the obtained fusion characteristics into a generation network of the high-resolution visual signal to realize the reconstruction of the high-resolution visual signal;

3, training the HasR model designed in the step 2 by using a model optimization algorithm to finally obtain optimal model parameters for a subsequent testing stage;

and 4, inputting the paired low-resolution visual signals and tactile signals to be detected into an optimal HasR model, wherein the optimal HasR model is used for extracting and fusing the characteristics of the low-resolution visual signals and the corresponding tactile signals, and generating the required high-resolution visual signals by utilizing the fused characteristics.

2. The method for reconstructing the super-resolution image oriented to the cross-modal communication according to claim 1, wherein: the step 1 specifically comprises the following steps:

step 1-1: for training data sets

It contains paired tactile, low-resolution visual signals and high-resolution visual signals, N is the number of paired visual and tactile signals, d _i ＝{h _i ,l _i ,t _i H represents a high-resolution visual signal, a low-resolution visual signal and a corresponding tactile signal, respectively, and the ith high-resolution visual signal h _i Inputting a coding network of the high-resolution visual signal, extracting the coding features z of the visual signal _h ；

Step 1-2: encoding feature z of the obtained high-resolution visual signal _h Inputting the video signal into a decoder for generating a high-resolution visual signal formed by a countermeasure network, then inputting the high-resolution visual signal reconstructed by the decoder into a discriminator for the high-resolution visual signal, finally learning the coding characteristics of the high-resolution visual signal by jointly training the encoder and the decoder and optimizing by using reconstruction loss and discrimination loss, and a corresponding decoding network, wherein a defined loss function is as follows:

L _pre ＝L _rec +αL _pre-adv ,

where α is a coefficient used to adjust the ratio of different losses, the first term loss is the reconstruction loss:

where (C, H, W) is the size of the high resolution visual signal, G _h A decoding network representing a corresponding high resolution visual signal,

representing the coding characteristics of the high-resolution visual signal obtained by the corresponding coding network, | · | | calving | ₁ Represents the corresponding L ₁ Norm, the second term is the loss generated against the network, and the specific loss function is:

where E (—) represents the expected value of the distribution function, p (z) _h ) Representing the distribution of the coding features of the high-resolution image, p (h) representing the distribution of the true high-resolution image, D _h A discrimination network representing the corresponding high-resolution visual signal for performing a determination of the reconstructed high-resolution visual signal, theta _gh And theta _dh Parameters of the generator and the discriminator, respectively, representing the corresponding high-resolution visual signal, by minimizing L _pre And obtaining an optimal coding network of the high-resolution visual signal, a corresponding coding characteristic of the high-resolution visual signal, a corresponding decoding network and a corresponding discrimination network of the high-resolution visual signal.

3. The method for reconstructing the super-resolution image oriented to the cross-modal communication according to claim 1, wherein: the step 2 comprises the following steps:

step 2-1: preliminary feature extraction of low resolution visual signals and corresponding haptic signals based on a training data set

Paired low resolution visual signal l present in _i And a haptic signal h _i Completing the primary feature extraction of the low-resolution visual signal and the tactile signal by utilizing a deep neural network to obtain the coding feature f of the corresponding low-resolution visual signal _l And corresponding haptic signal coding feature f _t ；

Step 2-2: coding features f based on the obtained low resolution visual signal _l And corresponding haptic signal coding feature f _t Establishing a feature mapping network to effectively reduce the heterogeneity difference between the modes, learning the identification representation in the modes and the consistent representation between the modes from the same mode and the cross-mode, and finally obtaining mapping features which comprise mapping of the low-resolution visual signalsRadial feature z _l And a mapping characteristic z of the haptic signal _t Then inputting the obtained mapping characteristics into a channel model, and executing a corresponding characteristic fusion step after the main terminal receives the mapping characteristics;

cross-modal semantic relevance learning: selecting triple losses to carry out cross-modal semantic correlation learning, and achieving the following effect through the learning of a mapping network, namely that the distances between low-resolution visual signal features and tactile signal features from the same category are close, and the distances between low-resolution visual signal features and tactile signal features from different categories are far; the loss function is defined as follows:

wherein theta is _l And theta _t Parameters representing the corresponding mapping network of low resolution visual signals and haptic signals, respectively, p and q represent different categories, N represents the number of instances of the corresponding low resolution visual signals and haptic signals, σ represents a corresponding threshold value, and L represents ₂ ＝||·|| ₂ Represents the corresponding L ₂ Norm, the total loss of semantic relevance, is expressed as the sum of the two, i.e.:

discriminative learning within isomorphous: while guaranteeing the semantic relevance, the problem of semantic identification in the same modality is effectively solved, namely, for samples in the same modality, the samples belonging to the same category are closer, and the samples belonging to different categories are farther, and the method is completed by adding a common classifier after a network is mapped, wherein the specific loss is represented as follows:

wherein p is _i (z) probability distribution representing classifier prediction, y _i Is a true tag, θ _c Parameters representing respective common classifiers; after the above processing, the mapping characteristic z of the obtained low-resolution visual signal is obtained _l And a mapping characteristic z of the haptic signal _t Inputting a channel model after normalization;

step 2-3: after transmission through the channel, the mapping characteristic z of the corresponding noisy low resolution visual signal is obtained at the master terminal _l-n And a mapping characteristic z of the haptic signal _t-n The ability to generate a fitted data distribution of the countermeasure network is used to select it to perform the feature fusion task, where z _h Regarded as a true sample, z _l-n And z _t-n Viewed as an input to the generator, z _m Representing the obtained fusion characteristics, the loss of the defined fusion network is as follows:

L _m ＝L _m-adv (G _m ,D _m )+L ₂ (z _m ,z _h )，

wherein the first term represents the loss of the ordinary generative countermeasure network, G _m Representing a feature fusion network, D _m Representing a feature authentication network corresponding to the feature fusion network, which is specifically represented as:

where E (#) represents the expected value of the distribution function, p (z) _h ) Distribution, p (z), representing coding characteristics of high-resolution images _m ) Distribution, θ, representing fusion characteristics _gm Representing the corresponding parameter of the converged network generator, θ _dm Representing model parameters of the authentication network corresponding to the converged network generator, the second term representing L ₂ Loss, facilitating maintaining semantic consistency；

Step 2-4: after the fusion characteristics are obtained, the reconstruction of the high-resolution visual signal is realized by using the fusion characteristics, the decoding network and the corresponding identification network of the high-resolution visual signal obtained in the step 1 are used, the network structure and the parameters of the decoding network and the corresponding identification network are used as the initialization network parameters of the step, and L is lost in the step 1-2 _pre On the basis, the perception loss is added to make the generated visual signal more consistent with the perception characteristic of human, and the loss is specifically expressed as follows:

L _finet ＝L _per +βL _adv-finet +γL _rec ，

wherein the first term is the perceptual loss, which is specifically expressed as:

wherein M is _i,j Number of parameters representing corresponding characteristic map, F ^i,j Represents the output after the convolution layer of the ith layer and before the maximum pooling layer of the jth layer of the VGG-19 network, beta and gamma are hyper-parameters, and the second term is the loss of the generation network of the high-resolution visual signal, which is expressed in detail as

Wherein theta is _gh Representing the corresponding utilization fusion feature z _m Parameters of the generation network, θ, for high resolution visual signal reconstruction _dh Representing the parameters of the corresponding authentication network, and the third term is the reconstruction loss.

4. The method for reconstructing the super-resolution image oriented to the cross-modal communication according to claim 1, wherein: the step 3 specifically comprises the following steps:

step 3-1: training an encoding network and a corresponding decoding network of the high-resolution visual signal and a corresponding discrimination network of the high-resolution visual signal by using the existing high-resolution visual signal;

step 3-2: based on the step 3-1, obtaining a coding network of the high-resolution visual signal, a decoding network of the high-resolution visual signal and a corresponding identification network, training the coding and mapping network of the corresponding low-resolution visual signal, the coding and mapping network of the haptic signal and the feature fusion network by using the low-resolution visual signal and the corresponding haptic signal, and finely adjusting the generating network of the corresponding high-resolution visual signal, wherein the coding network of the low-resolution visual signal and the coding network of the haptic signal are obtained by loading the pre-trained networks, and do not participate in updating in the step.

5. The method for reconstructing the super-resolution image oriented to the cross-modal communication according to claim 4, wherein: the specific process of the step 3-1 is as follows:

step 311, initializing parameter θ _eh (0)，θ _gh (0)，θ _dh (0) The value of the corresponding parameter for iteration 0;

step 312, setting the iteration number to n ₁ Learning rate of mu ₁ ；

Step 313, optimizing network parameters by using a RMSProp algorithm:

wherein theta is _eh (n+1)，θ _dh (n+1)，θ _gh (n + 1) and θ _eh (n)，θ _dh (n)，θ _gh (n) coding network of high resolution visual signals corresponding to the (n + 1) th and nth times, respectively, high resolution visualA discrimination network of signals, parameters of a generation network of high resolution visual signals,

making partial derivatives for each loss function;

step 314, if n<n ₁ Then repeat step 313 through n ₁ After iteration, a network converging to the optimal is obtained, and the network comprises a coding network of the high-resolution visual signal, an identification network of the high-resolution visual signal and a generation network of the high-resolution visual signal.

6. The method for reconstructing the super-resolution image oriented to the cross-modal communication according to claim 4, wherein: the specific process of the step 3-2 is as follows:

step 321, initializing parameter theta _l (0)，θ _t (0)，θ _c (0) Representing initial random parameters of the corresponding network, and loading the parameters of the generation network and the discrimination network of the high resolution visual signal obtained in step 3-1 as theta _gh (0)，θ _dh (0) The initial parameters of (a);

322, starting iteration, and setting the iteration number as n ₂ The learning rate is mu ₂ ；

Step 323, optimizing parameters of a low-resolution visual signal mapping network, a tactile signal mapping network and a public classifier by adopting an Adam algorithm:

wherein theta is _c (n+1)，θ _l (n+1)，θ _t (n + 1) and θ _c (n)，θ _l (n)，θ _t (n) parameters of the n +1 th and nth corresponding common classifiers, respectively, a low resolution visual signal mapping network and a corresponding haptic signal mapping network,

making partial derivatives for each loss function;

step 324, optimizing the parameters of the feature fusion network and the corresponding identification network of the fusion features by adopting an Adam algorithm:

wherein theta is _gm (n+1)，θ _dm (n+1)，θ _gm (n) and θ _dm (n) parameters of the n +1 th and nth corresponding fused feature generation networks and fused feature authentication networks, respectively,

making partial derivatives for each loss function;

step 325, fine-tuning the generation network and the identification network of the high-resolution visual signal by adopting a RMSProp algorithm, and optimizing and updating through the following functions:

wherein theta is _gh (n+1)，θ _dh (n+1)，θ _gh (n) and θ _dh (n) parameters corresponding to the generation network and the discrimination network of the high-resolution visual signal respectively corresponding to the (n + 1) th and nth times,

making partial derivatives for each loss function;

step 326, if n<n ₂ Then go to step 323, go through n ₂ After iteration, the optimal HasR network converged to is obtained, and the optimal HasR network comprises a mapping network of low-resolution visual signals, a mapping network of tactile signals, a feature fusion network and a corresponding identification network of fusion features, a fine-tuned identification network converged to the optimal high-resolution visual signals and a high-resolution visual signal generation network.

7. The method for reconstructing the super-resolution image oriented to the cross-modal communication according to claim 1, wherein: the step 4 comprises the following steps:

step 4-1: adopting a trained HasR model;

step 4-2: and inputting a group of paired low-resolution visual signals and corresponding tactile signals into a HasR model, completing the coding, mapping and fusion of modal characteristics, and finally obtaining the corresponding high-resolution visual signals.

Images