CN114782992A - Super-joint and multi-mode network and behavior identification method thereof - Google Patents

Abstract

The invention relates to the technical field of neural networks, in particular to a super-joint and multi-mode network and a behavior identification method thereof, which comprises the following steps: collecting a human body depth map, extracting features of the depth map by using a DMMS flow, and calculating depth data prediction scores; collecting a human body skeleton sequence, respectively extracting original joint and super joint data, combining super joints and common joints to construct skeleton information, sending the skeleton information into a structured space-time characteristic learning model to obtain static and dynamic joint data streams and static and dynamic super joint data streams, and performing self-adaptive weight fusion on the original joint and super joint data streams to respectively obtain joint data prediction scores and super joint prediction scores; and adding the classification prediction scores of the DMMS flow and the skeleton flow to generate a final prediction score. The invention learns the abundant texture information of human body parts in space from the depth map and learns the abundant space-time characteristics in the motion posture change from the skeleton sequence.

Description

A hyper-joint and multimodal network and its method in behavior recognition

technical field

The invention relates to the technical field of neural networks, in particular to a hyper-joint and multi-modal network and a method for recognizing it in action.

Background technique

Human action recognition has become a hot topic in the field of machine learning and computer vision because of its broad application prospects, and has very important theoretical research value. Different from the recognition of static images, behavior recognition should comprehensively consider continuously changing images, and construct the spatiotemporal feature representation of human behavior from a series of motion sequences composed of static human pose images to realize the classification and recognition of actions. At present, human behavior recognition has been widely used in assisting human-computer interaction, sports analysis, intelligent monitoring and virtual reality and other fields.

At present, behavior recognition algorithms are mainly divided into two categories:

(1) Behavior recognition based on traditional methods. In traditional methods, scholars extract handcrafted features as effective spatiotemporal feature descriptors. Donglu Li et al. combined the characteristics of depth data with point cloud data and used depth information to reconstruct 3D point cloud space to improve the spatial information expression ability of depth data. Then they evenly divided the point cloud space into many grid spaces, and calculated the proportion of the number of point clouds in the grid space as a spatial distribution feature for action recognition. Liu et al. established distribution sectors on the 2D projection plane of 3D skeleton joint trajectories, and then used histograms to count the distribution sectors of specific projections as skeleton-based motion descriptor HDS-SP to characterize spatial and temporal information.

(2) Behavior recognition based on deep learning. The method of action recognition based on extracting handcrafted features has high complexity in implementation. Traditional methods require feature engineering, and the analysis and design of motion features greatly affect the classification accuracy. In the action recognition method based on deep learning, the feature learning network is first used to learn the feature representation of the action sequence, and then the features are sent to the classification network for classification. Shengquan Wang et al. proposed a multi-level deep feature fusion enhancement network (MDFFEN), which firstly extracted spatiotemporal features from sub-modules of the two-stream network to form multi-depth features, and then fused these spatiotemporal features for classification. Haoran Wang et al. proposed the Skeleton Edge Motion Network (SEMN), which mines the motion information of the human body on the skeleton edge motion, and combines the angle change of the skeleton edge with the corresponding joint motion to determine the motion of the skeleton edge.

Although depth images carry dense texture information, the rich texture information also brings high information redundancy, making them insensitive to local pose changes of human joints. Human skeleton data has become a hot research object in the field of human action recognition due to its high sensitivity to pose changes. Compared with depth image data, the three-dimensional spatial structure information provided by skeleton data can greatly reduce the influence of factors such as height, body contour, and clothing of the data collection object. Although the skeleton joints are specially designed to capture the spatial structure information of the human body, the information carried by the joint points can only represent the position but not the dependencies with the adjacent points. Previous methods usually use the position of the joint as the spatial structure information representation of the joint, while ignoring the dependencies between the joints.

SUMMARY OF THE INVENTION

In view of the shortcomings of the existing algorithms, the present invention proposes a human behavior recognition method based on skeleton dependencies and multi-modal spatiotemporal feature representation, which learns the rich texture information of human body parts in space from the depth map, and learns from the skeleton sequence. Rich spatiotemporal features in motion pose changes.

The technical scheme adopted by the present invention is: a hyper-joint and multi-modal network and its behavior recognition method include the following steps:

S1. Collect human body depth data, input the DMMS stream to perform feature extraction on the depth map, and calculate the DMMs stream prediction score;

S2. Collect the human skeleton sequence, extract the original joint and hyper-joint data respectively, combine the hyper-joint and ordinary joint to construct skeleton information, calculate the static and motion data of the hyper-joint and ordinary joint respectively, and send it into the structured spatiotemporal feature learning model to obtain the static state and dynamic joint data streams, static and dynamic hyper-joint data streams, feature-level fusion of static and motion data from common joints and hyper-joints, respectively, and calculation of prediction scores for common and hyper-joints.

S3. First, perform adaptive weight fusion on the classification scores of the original joint and the super-joint to obtain the prediction score of the skeleton flow;

S21 , constructing super-joints according to the dependencies of common joints, and constructing skeleton information by combining super-joints and common joints;

Further, the hyperjoint construction includes:

First, calculate RWE and RWH, which are the direction vectors of RW pointing to RE and RH, respectively. The calculation formula is as follows:

Among them, RW, RE and RH represent the right wrist, right elbow and right hand in the human skeleton, respectively;

Then, the Cartesian product of the two intersecting vectors is calculated to obtain the normal vector n of the plane where the vectors are located. The calculation formula is as follows:

Then, using the direction vectors of the two bones with RE as the starting point, calculate the angle between the two direction vectors. The calculation formula is as follows:

Finally, the hyperjoint data vector HyperJoint is obtained, and the formula is:

Further, constructing skeleton information by combining hyper-joints and common joints includes the following steps:

The skeleton diagram that constitutes the human body is represented by the variable G(V,H), and the formula is:

Among them, V is the set of spatial nodes that constitute the human skeleton graph, and H is the dependency established on the joints of the human skeleton;

The dynamic information of the skeleton is defined as the difference between the skeleton elements on the adjacent skeletons on the skeleton sequence, and establishes a connection to the skeleton pose in the time domain. The formula is as follows:

S22, sending the skeleton information into the structured spatiotemporal feature learning model, including the following steps:

The spatiotemporal feature learning module of convolutional neural network is used to extract human action features from skeleton information;

Further, the spatiotemporal feature learning module includes a node timing feature learning block and a spatial global feature learning block. By stacking the spatiotemporal blocks, a feature learning module is constructed to learn the effective spatiotemporal features of the original skeleton nodes and the dependencies between the nodes from the skeleton data;

Further, the construction of the node timing feature learning block includes the following steps:

First, use the convolution block attention module to increase attention to the tensors of the input network, which means to improve the feature expression ability of the input data;

Then, the location features of the skeleton joints are automatically learned in the network using a convolutional layer with a kernel size of (1×1); the formulation is defined as:

f _T (X)=σ(φ(Attn(X))) (9)

φ表示由卷积层结构的函数，σ表示ReLU激活函数，而Attn表示一个注意力区块；where X represents a third-order tensor

φ represents the function structured by the convolutional layer, σ represents the ReLU activation function, and Attn represents an attention block;

通过如下公式进行定义：Finally, a convolution kernel of size (3×1) is used to aggregate the learned feature information on the skeleton joint positions in time, and the output is a third-order tensor

It is defined by the following formula:

A=φ(f _T (X)) (10).

Further, the construction of the spatial global feature learning block includes:

A convolution kernel of size (3×3) is used to automatically learn the semantic features of the coordinated movement of skeleton nodes in the spatial domain on the network. The convolution operation can aggregate the spatial structure information of the elements constituting the human body, which is formulated as:

f _S (A)=φ(A') (11)

变换成

Among them, A' represents the transformation of the matrix, and a third-order tensor

transform into

S3. Add the classification prediction scores of the DMMS flow and the skeleton flow to generate the final prediction score;

Beneficial effects of the present invention:

1. The skeleton node dependency relationship is proposed to re-model the spatial structure information representation of the human skeleton sequence. The isolated nodes and adjacent points that constitute the human skeleton are established to display the dependency relationship, and the human skeleton sequence and the skeleton node dependency sequence are used as the network. Input for spatiotemporal feature learning.

2. A new spatiotemporal feature learning network is designed to learn spatiotemporal features of skeleton sequences and skeleton node dependencies; in the network, a node timing feature learning block (JTFLB) is constructed to explore the temporal relationship of elements on the human skeleton. Due to the holistic and synergistic characteristics of human motion, we construct a Spatial Global Feature Learning Block (SGFLB) to aggregate the semantic features learned in the temporal domain from the elements constituting the human skeleton.

3. Synergistically extract rich spatiotemporal features from the depth sequence and skeleton sequence of the human body, and improve the performance of action recognition by fusing the scores of different modalities; in order to verify the effectiveness of the present invention, experiments are carried out on public datasets of different scales. Evaluation, compared with single modality, the recognition effect is significantly improved after multimodal fusion.

Description of drawings

Fig. 1 is the super-joint and multimodal network of the present invention and its structural block diagram in the behavior recognition method;

Fig. 2 is the schematic diagram of calculating joint posture direction of the present invention;

3 is a schematic diagram of calculating a joint angle of the present invention;

Fig. 4 is the establishment super-joint schematic diagram of the present invention;

Fig. 5 is the joint timing feature learning block diagram of the present invention;

Fig. 6 is the spatial global feature learning block diagram of the present invention;

Fig. 7 is the frame number statistics of UTD-MHAD of the present invention and NTU RGB+D data set;

Fig. 8 is the difference between the basic model of the present invention and the fusion model on UTD-MHAD;

Fig. 9 is the difference between the basic model of the present invention and the fusion model on NTU RGB+D CS;

FIG. 10 is the difference between the basic model of the present invention and the fusion model on NTU RGB+D CV.

Detailed ways

The present invention will be further described below with reference to the accompanying drawings and embodiments. This figure is a simplified schematic diagram, and only illustrates the basic structure of the present invention in a schematic manner, so it only shows the structure related to the present invention.

In order to evaluate the effectiveness of the method of the present invention, experiments are carried out on a public dataset based on the depth map and skeleton information; a large public dataset can provide a wider range of training data for the model, making the model stronger; in order to verify the robustness of the method of the present invention Due to the nature of the dataset, a classic small dataset is used in the selection of the dataset. Therefore, experiments are conducted on several datasets with distinct scales: UTD-MHAD and NTU-RGB+D.

The invention is established based on the PyTorch framework, wherein the Python version is 3.7.0, and the pytorch version is 1.10.1; the hardware platform of this experiment is a desktop computer, wherein the main board is MSI B460M MORTAR, the CPU is Intel i710700, and the main frequency is 2.9 GHz, the memory is 16GB, the operating system is Windows 10 Professional Edition, the GPU resource is NVIDIA TeslaV100, and the video memory is 32GB; the software tools used in the experiment are PyCharm and Anaconda3.

The UTD-MHAD dataset includes a total of 861 action samples, and each action sample is provided with four kinds of data, RGB, depth map, skeleton sequence, and inertial sensor sequence captured from a viewpoint using a fixed-position acquisition device; including 27 action categories , each action class was repeated 3-4 times by 8 subjects; in order to facilitate the comparison with the prior art, the action sequences of subjects numbered 1, 3, 5, and 7 were used as the training set , and the rest are used as the test set.

NTU RGB+D belongs to a large dataset, and provides more challenging action samples and more modal information; NTU RGB+D dataset includes a total of 56880 action samples, and uses fixed different location acquisition devices for each The sample provides four kinds of data, RGB, depth map, skeleton sequence and infrared radiation video captured from three viewpoints; includes 60 action categories, each action category is completed 1-2 times by 40 subjects; collected with UTD-MHAD In different ways, NTU RGB+D provides 17 kinds of multi-view and multi-modal data collected by acquisition equipment at different levels and distances; two standard experimental strategies are used in the data set, which are divided into cross-subject and cross- Perspective; in a cross-subject strategy, all subjects are first divided into two groups, where subjects are numbered 1, 2, 4, 5, 8, 9, 13, 14, 15, 16, 17, 18 , 19, 25, 27, 28, 31, 34, 35 and 38 action sequences are used as training set and the rest are used as test set. In the cross-perspective strategy, the cameras with different perspectives in the acquisition device are firstly divided into two groups, in which the 37920 action sequences collected by the cameras numbered 2 and 3 are used as the training set, and the 18960 action sequences collected by the camera number 1 are used as the test set. .

As shown in Figure 1, a hyper-joint and multimodal network and its behavior recognition method, including the following steps:

There are two flows in Fig. 1, namely the upper half of Fig. 1: DMMs flow and the lower half of Fig. 1: skeleton flow, Fig. 1(a) calculates the static and motion data of original joints and super-joints respectively; Fig. 1(b) passes Stacking JTFLB and SGFLB builds a structured spatiotemporal feature learning module to learn spatiotemporal feature representations from skeleton sequences; Figure 1(c) performs feature-level fusion of static and motion data from joints and hyper-joints, respectively; The prediction scores of the original joint and super-joint are fused with adaptive weights, and then the prediction scores of the two streams are added to generate the final prediction score;

S1 collects the depth map of the human body, uses the DMMS stream to perform feature extraction on the depth map, and calculates the depth data prediction score;

S2. Collect the human skeleton sequence, extract the original joint and hyper-joint data respectively, and send it to the structured spatiotemporal feature learning model to obtain static and dynamic joint data streams, static and dynamic hyper-joint data streams, respectively, and combine the original joint and hyper-joint data. Adaptive weight fusion is performed on the data stream of the data stream, and the prediction score of joint data and the prediction score of hyper-joint are obtained respectively;

它表示人体骨架的空间结构在时间维度t＝1,2,...,T上的变化；然而，构成人体骨架的关节仅携带位置信息，因此，在人体中自然连接的关节间建立依赖关系去获得关节位置以外的高级信息。Further, given a human skeleton sequence {S ₁ , S ₂ ,..., S _T } representing the motion process, an action descriptor is calculated, which can better describe the change of the action; the human skeleton sequence is regarded as a function

It represents the change of the spatial structure of the human skeleton in the time dimension t=1,2,...,T; however, the joints that make up the human skeleton only carry position information, so a dependency relationship is established between the joints that are naturally connected in the human body to get advanced information beyond joint position.

In the human behavior represented by the skeleton sequence, joints are the main moving parts. As shown in Figure 2, RW, RE and RH represent the right wrist, right elbow and right hand in the human skeleton, respectively. Taking the movement of the right wrist joint as an example, the coordinates of RW, RE and RH determine a plane in space, the normal vector of the plane determines the direction, and the normal vector of the plane is calculated as the direction descriptor of the local posture centered on the right wrist joint;

First, calculate RWE and RWH, which are the direction vectors of RW pointing to RE and RH, respectively. The calculation formula is as follows:

Then, the Cartesian product of the two intersecting vectors is calculated to obtain the normal vector n of the plane where the vectors are located. The calculation formula is as follows:

For different actions, the range of bone movement is different. As shown in Figure 3, when the right wrist joint moves, the angle between the two bones connected by the joint will also change. The included angle serves as a descriptor of the magnitude of motion;

First, we need to calculate the direction vector of the two bones with RE as the starting point, as shown in formula (2); then, calculate the angle between the two vectors, and the calculation formula is as follows:

Since the human body local pose descriptor and the human body local limb motion descriptor are obtained by aggregating the information of adjacent joints, they are combined together to form a joint dependency relationship; the joint dependency relationship corresponds to a specific joint, in order to distinguish it from the joint position, it will represent the joint The vector of dependencies is called HyperJoint, and the calculation of HyperJoint is as follows:

原始骨架序列仅仅表示了人体骨架关节位置在时间维度t＝1,2,...,T上的变化，而HyperJoint中聚合了来自邻接点的信息，表示局部姿态方向和局部肢体运动幅度。The constructed HyperJoint is significantly different from the joints in the skeleton data, and the skeleton sequence formed by the HyperJoint is regarded as a function

The original skeleton sequence only represents the changes of the joint positions of the human skeleton in the time dimension t=1, 2, ..., T, while the information from the adjacent points is aggregated in HyperJoint, which represents the local pose direction and the local limb motion range.

As shown in Figure 4, the information carried by common joint points is used to describe the position of joints in three-dimensional space, that is, joint information does not include the multivariate relationship between joint points; the relationship between objects in reality is often complex The multivariate relationship; therefore, if the univariate relationship carried by the common joint points is converted into a multivariate relationship, then a lot of useful information will be generated; the difference between HyperJoint and common joint points is not only the dimension of the joint points. In terms of joints, HyperJoint more accurately describes the relationship between multiple objects that are related.

S21, constructing super-joints according to the dependencies of common joints, and constructing skeleton information according to super-joints and common joints;

The skeleton graph that constitutes the human body is represented by the variable G(V, H), where V is the set of spatial nodes (common joints) that constitute the skeleton graph of the human body, and H is the dependency (hyper-joint) established on the human skeleton joints; node information V and dependency H are expressed by the following formula:

The dynamic information of the skeleton is defined as the difference between the skeleton elements on the adjacent skeletons on the skeleton sequence, and establishes a connection to the skeleton pose in the time domain. The formula is as follows:

S22. A structured spatiotemporal feature learning model for skeleton data;

In the natural environment, human behavior is dynamic and continuous; however, in the acquisition device, the complete human behavior is divided into a series of human poses on the time axis, and human poses can be further decomposed into three-dimensional space. For human skeleton points, since a continuous action shows a hierarchical structure from local to whole in the process of forming, the spatiotemporal feature learning module of convolutional neural network is used to extract human action features from skeleton data;

The feature learning module consists of two main functional blocks, which are divided into node timing feature learning block and spatial global feature learning block; among them, the node timing feature learning block deals with the timing features of the skeleton nodes, and the spatial global feature learning block deals with the Spatial information distribution features; by stacking spatiotemporal blocks, a feature learning module is constructed to learn effective spatiotemporal features of dependencies between original skeleton nodes and hyperjoint nodes from skeleton data;

其中，N表示构成人体骨架的元素个数，C表示由元素维度构成的通道数量，T表示输入动作的帧数。Further, the relationship between elements on the human skeleton in terms of time sequence is learned through the node timing feature learning block (JTFLB). The node timing feature learning block is shown in Figure 5. The node timing feature learning block is to decompose the human body skeleton into a series of basic Component elements, each component element has its own trajectory when moving, and these trajectories have semantic features in the time domain. For the node timing feature learning block, the input data is a third-order tensor

Among them, N represents the number of elements constituting the human skeleton, C represents the number of channels composed of element dimensions, and T represents the number of frames of input actions.

First, the convolution block attention module is used to add attention to the tensors of the input network to improve the feature expression ability of the input data; among them, the convolution attention module is composed of a channel attention module and a spatial attention module; then the convolutional attention module is used. A convolutional layer with a kernel size of (1×1) automatically learns the positional features of the skeleton joints in the network; it is formulated as:

f _T (X)=σ(φ(Attn(X))) (9)

φ表示由卷积层结构的函数，σ表示ReLU激活函数，而Attn表示一个注意力区块。where X represents a third-order tensor

φ represents the function structured by convolutional layers, σ represents the ReLU activation function, and Attn represents an attention block.

通过如下公式进行定义：Finally, a convolution kernel of size (3×1) is used to aggregate the feature information learned from the skeleton joint positions in time, and the output is a third-order tensor

It is defined by the following formula:

A=φ(fT(X)) (10)

其中，N是构成人体骨架的元素个数，C表示通道数量，T表示输入动作的帧数；Further, the spatial global feature learning block (SGFLB) is used to learn the relationship between the elements on the human skeleton in motion, as shown in Figure 6, the spatial global feature learning block is to participate in the spatial learning of different parts of the human body movement, because the human body movement The integrity and cooperativity of , have spatial semantic features; for the spatial global feature learning block, the input data is a third-order tensor

Among them, N is the number of elements constituting the human skeleton, C is the number of channels, and T is the number of frames of input actions;

A convolution kernel of size (3×3) is used to automatically learn the semantic features of the coordinated movement of skeleton nodes in the spatial domain on the network. The convolution operation can aggregate the spatial structure information of the elements constituting the human body, which is formulated as:

f _S (A)=φ(A') (11)

变换成

Among them, A' represents the transformation of the matrix, and a third-order tensor

transform into

S3. Add the classification prediction scores of the DMMS flow and the skeleton flow to generate the final prediction score;

To study the effect of the length of the skeleton sequence on the recognition rate, in Figure 7 is the statistical histogram of the number of frames using the two scale datasets; Figure 7(a) is the statistical histogram of the number of frames in the UTD-MHAD dataset, from Figure 7 It can be seen in 7(a) that the frame number distribution interval of the UTD-MHAD data set is [40,125]; Figure 7(b) is the main frame number statistical histogram of the NTU RGB+D data set, from Figure 7(b) It can be seen that the main frame number distribution interval of the NTU RGB+D data set is [25, 200]; in order to effectively explore the impact of the frame rate on the experimental results, the number of frames is divided into three levels: 32, 64 and 128. Because UTD-MHAD It is a small-scale data set, so an additional experiment when the number of frames is 16 is added; the experimental results are shown in Table 1. In this table, we can clearly see the change of the recognition rate corresponding to the selection of skeleton sequences of different lengths.

Table 1 Influence of the number of frames on the classification performance

As the number of frames increases, the recognition rate on the two datasets changes significantly. In the UTD-MHAD dataset, the recognition rate on CS is only 43.35% when the number of frames is 16. As the number of frames increases to 32, the recognition rate is increased to 77.67%; when the number of frames is less than 32, the information provided to the model in the frame number interval with the center frame number of 16 is not enough to express the entire action; when the number of frames increases from 32 to 64 , the recognition rate dropped to 76.74%; the action frames added in this interval failed to provide discriminative information to the model, on the contrary, the redundancy caused by the increase of action frames resulted in a decrease in accuracy; as the number of frames increased from 64 By 128, the recognition rate has risen to 91.16%. Therefore, in the UTD-MHAD dataset, due to the large redundancy in the middle part of the action sequence, when enough frames are obtained, the model will be more discriminative;

In the NTU RGB+D dataset, the recognition rates of the two experimental strategies also increase with the increase of the number of frames, so the higher number of frames does bring more discriminative information to the model; in order to ensure the accuracy of the model parameters For consistency, 128 frames are sampled from both datasets to verify the robustness of the method of the present invention on large-scale data.

The effect of a single modality in classification is limited, so the depth sequence and the skeleton sequence are fused to improve the recognition accuracy; Table 2 shows the recognition accuracy before and after fusion, and the method of the present invention has significantly improved recognition accuracy in both datasets; In order to facilitate the analysis of the improvement effect of the fusion depth feature, Figures 8-10 list the variation of the accuracy of the method of the present invention on the specific actions of the two data sets.

In Figure 8, the method of the present invention has improved the recognition accuracy of each action category in the UTD-MHAD data set; among them, the action numbered 6 has the greatest improvement after fusion of depth features, and its recognition accuracy is increased by nearly 30%. In the actions to improve the recognition accuracy, the recognition accuracy of most actions is improved by about 5%; it is found in FIG. Very good recognition accuracy.

The method of the present invention also has a good effect on the NTU RGB+D data set. Figure 9 shows the results of the experimental strategy CS. The model pairs of the present invention are numbered 1, 10, 11, 12, 13, 16, 17, and 20 The recognition accuracy of the actions has been significantly improved; Figure 10 shows the results of the experimental strategy CV, and the fusion model has significantly improved the recognition accuracy of actions numbered 4, 11, 12, 28, 29, and 30; Taken together, the fused model improves the accuracy by 6.76% on CS and 3.28% on CV.

Table 2 Multimodal performance

The method of the present invention is compared with the method of the prior art on the UTD-MHAD data set, and the results are as shown in Table 3:

Table 3 is compared with the prior art method on the UTD-MHAD dataset

The results of the method of the present invention and other methods on UTD-MHAD are shown in Table 3, the method of the present invention has the highest recognition rate; other methods in Table 3 are divided into methods based on handcrafted features and methods based on deep learning; and BayesianGC Compared with the LSTM method, the accuracy is improved by 4.64%; compared with the network based on the skeleton edge, the accuracy is improved by 1.15%.

The present invention establishes a display dependency between independent joints and adjacent points, and this feature can be used to describe the local posture and motion range in the process of human behavior; in order to better learn the semantic features of the skeleton sequence in space and time, We design two functional modules, JTFLB and SGFLB. JTFLB learns the temporal features of skeleton nodes, and SGFLB aggregates the temporal features learned from skeleton nodes in the spatial domain.

Taking the above ideal embodiments according to the present invention as inspiration, and through the above description, relevant personnel can make various changes and modifications without departing from the technical idea of the present invention. The technical scope of the present invention is not limited to the contents in the specification, and the technical scope must be determined according to the scope of the claims.

Claims (8)

1. a super-joint and a multimodal network and a method for recognizing behavior thereof, are characterized in that, comprise the following steps: S1 collects the depth map of the human body, uses the DMMS stream to perform feature extraction on the depth map, and calculates the depth data prediction score; S2. Collect the human skeleton sequence, extract the original joint and hyper-joint data respectively, combine the hyper-joint and ordinary joints to construct the skeleton information, send the skeleton information into the structured spatiotemporal feature learning model, and obtain the static and dynamic joint data flow, static and Dynamic super-joint data flow, and fuse the data flow of original joint and super-joint with adaptive weights to obtain joint data prediction score and super-joint prediction score; S3. Add the classification prediction scores of the DMMS flow and the skeleton flow to generate a final prediction score. 2. super-joint and multimodal network according to claim 1 and its method for recognizing in behavior, it is characterized in that, described step S2 comprises: S21, constructing super-joints according to the dependencies of common joints, and constructing skeleton information according to super-joints and common joints; S22, sending the skeleton information into the structured spatiotemporal feature learning model. 3. super-joint and multimodal network according to claim 2 and its method for recognizing in behavior, it is characterized in that, described super-joint construction comprises: First, calculate RWE and RWH, which are the direction vectors of RW pointing to RE and RH, respectively. The calculation formula is as follows:

Among them, RW, RE and RH represent the right wrist, right elbow and right hand in the human skeleton, respectively; Then, the Cartesian product of the two intersecting vectors is calculated to obtain the normal vector n of the plane where the vectors are located. The calculation formula is as follows:

Then, using the direction vectors of the two bones with RE as the starting point, calculate the angle between the two direction vectors. The calculation formula is as follows:

Finally, the hyperjoint data vector HyperJoint is obtained, and the calculation formula is:

4. The super-joint and multimodal network according to claim 3 and the method for recognizing the behavior thereof, wherein the construction of the skeleton information in combination with the super-joint and the common joint comprises: The skeleton diagram that constitutes the human body is represented by the variable G(V,H), and the formula is:

Among them, V is the set of spatial nodes that constitute the human skeleton graph, and H is the dependency established on the joints of the human skeleton; The dynamic information of the skeleton is defined as the difference between the skeleton elements on the adjacent skeletons on the skeleton sequence, and establishes a connection to the skeleton pose in the time domain. The formula is as follows:

5. The hyper-joint and multimodal network according to claim 2 and the method for recognizing the behavior thereof, wherein the sending the skeleton information into the structured spatiotemporal feature learning model comprises: utilizing the spatiotemporal feature of the convolutional neural network The learning module is used to extract human action features on the skeleton information. 6. The super-joint and multimodal network according to claim 5 and the method for recognizing it in action, wherein the spatiotemporal feature learning module comprises a node timing feature learning block and a spatial global feature learning block, and the spatiotemporal blocks are stacked by stacking the space-time feature learning block. , build a feature learning module to learn effective spatiotemporal features of dependencies between original skeleton nodes from skeleton information. 7. The super-joint and multimodal network according to claim 6 and the method for recognizing the behavior thereof, wherein: the construction of the node timing feature learning block comprises the steps: First, use the convolution block attention module to increase attention to the tensors of the input network, which means to improve the feature expression ability of the input data; Then, the location features of the skeleton joints are automatically learned in the network using a convolutional layer with a kernel size of (1×1); the formulation is defined as: f _T (X)=σ(φ(Attn(X))) (9) where X represents a third-order tensor

φ represents the function structured by the convolutional layer, σ represents the ReLU activation function, and Attn represents an attention block; Finally, a convolution kernel of size (3×1) is used to aggregate the learned feature information on the skeleton joint positions in time, and the output is a third-order tensor

It is defined by the following formula: A=φ(f _T (X)) (10) 8 . The hyper-joint and multimodal network and the method for recognizing the behavior thereof according to claim 6 , wherein the construction of the spatial global feature learning block comprises: using a convolution kernel with a size of (3×3) The semantic features of the coordinated movement of skeleton nodes in the spatial domain are automatically learned on the network, and the convolution operation can aggregate the spatial structure information of the elements that constitute the human body. The formula is defined as: f _S (A)=φ(A') (11) Among them, A' represents the transformation of the matrix, and a third-order tensor

transform into

Images