CN110647803B - A gesture recognition method, system and storage medium - Google Patents

Abstract

The application discloses a gesture recognition method which is applied to a gesture recognition system comprising a first radar and a second radar, and comprises the steps of judging whether gesture actions exist in radar data; if the gesture motion exists in the preset range, acquiring first gesture information and second gesture information; the first gesture information and the second gesture information correspond to gesture information acquired by the first radar and the second radar respectively; calculating to obtain a feature matrix according to the first gesture information and the second gesture information; the feature matrix comprises a first feature matrix and a second feature matrix corresponding to the first gesture information and the second gesture information; acquiring a gesture characteristic value according to the convolutional neural network; and recognizing the gesture according to the gesture value characteristics. The application further provides a gesture recognition system and a storage medium. According to the application, gesture recognition is carried out through the two radars at the same time, so that the types of recognized gestures can be increased, and the accuracy of gesture recognition is improved.

Description

A gesture recognition method, system and storage medium

Technical field

The invention relates to the field of computers, and in particular to a gesture recognition method, system and storage medium.

Background technique

Gesture recognition technology is an important topic in the field of human-computer interaction. The purpose is to obtain hand movement information through sensors and identify gesture types through algorithms. Gestures are an important part of communication in human daily life. They can intuitively and effectively convey the information people want to express in certain scenarios. Moreover, gesture movements are rich in types and have specific meanings, and can be directly used as a method of human-computer interaction to complete the interaction between humans and computers. In addition, gesture recognition technology can be fully applied in all aspects of human life, such as helping deaf-mute people communicate normally, recognizing traffic police gestures in smart driving, controlling home work in smart homes, etc. In short, gesture recognition technology, as the most natural way of human-computer interaction, will play a huge role in future life. Gesture recognition can be thought of as a way for computers to understand human language, creating a richer bridge between machines and humans than raw text user interfaces or even graphical user interfaces. Existing gesture recognition technologies include vision-based gesture recognition technology, micro-point system-based gesture recognition technology, and radar-based gesture recognition technology. Radar-based gesture recognition technology has a high data refresh rate, can work in various environments, and is not affected by light, dust, etc. It also has non-contact and natural privacy protection functions, and will become a non-contact technology in the future. An irreplaceable technology in the field of gesture recognition. However, when existing radar-based gesture recognition technology uses gesture information that can be collected by single-station radar, it can recognize fewer types of gestures, which causes many inconveniences in practical applications.

Contents of the invention

The main purpose of the present invention is to provide a gesture recognition method, system and storage medium to increase the types of gestures recognized.

In order to achieve the above object, the present invention provides a gesture recognition method, which is applied to a gesture recognition system including a first radar and a second radar. The method includes:

Determine whether there is a gesture action in the acquired radar data; the radar data includes radar data of the first radar and the second radar;

If it is determined that there is a gesture action within the preset range, first gesture information and second gesture information are obtained; wherein the first gesture information and the second gesture information correspond to the first radar and the second radar respectively. Collected gesture information;

According to the first gesture information and the second gesture information, a gesture feature matrix is calculated and obtained; wherein the gesture feature matrix includes the first feature matrix and the second feature corresponding to the first gesture information and the second gesture information. matrix;

Obtain gesture feature values according to the gesture feature matrix and convolutional neural network;

Recognize gestures based on the gesture feature values.

Further, the "determining whether there is a gesture action in the radar data" includes:

According to the radar data, detect whether there is a gesture action;

If there is a gesture action, establish plane coordinates with the first radar as the origin;

According to the plane coordinate information, determine whether the gesture action is within the gesture range;

If it is determined that the gesture action is within the gesture range, the first gesture information and the second gesture information are obtained according to the radar data; the gesture range is the common detection range preset by the first radar and the second radar.

Further, the "calculating and obtaining the gesture feature matrix based on the first gesture information and the second gesture information" includes:

Construct a first matrix and a second matrix respectively according to the first gesture information and the second gesture information;

Remove outliers in the first matrix and the second matrix;

The first matrix and the second matrix perform matrix flipping;

The first matrix and the second matrix after the matrix flip are completed to obtain the first characteristic matrix and the second characteristic matrix.

Further, the "removing outliers in the first matrix and the second matrix" includes:

According to the sliding window algorithm, the row data of the first matrix and the second matrix are smoothed.

Further, the "obtaining gesture feature values according to the gesture feature matrix and the convolutional neural network" includes:

Perform convolution and expansion on the first feature matrix and the second feature matrix respectively to obtain the first convolution and the second convolution;

Extend and connect the first convolution and the second convolution into one-dimensional gesture feature values;

According to the one-dimensional gesture feature value, the gesture is recognized.

Further, before "determining whether there is a gesture action within the preset range", the gesture recognition method also includes:

Remove the first radar or second radar receiving end's own interference signal;

Remove external interference signals.

Further, the "removing the first radar or second radar receiving end's own interference signal" includes:

Collect hand gesture-free signals at the second preset time;

Calculate the self-interference signal according to the non-gesture signal;

Remove the self-interference signal received by the first radar or the second radar.

Further, the "removing external interference signals" specifically includes removing external interference signals according to clutter suppression methods; the clutter suppression methods include: linear phase FIR filtering method, adaptive average clutter subtraction method, adaptive iterative clutter subtraction method, etc. Wave subtraction method.

The present invention also provides a gesture recognition system, including a first radar and a second radar. The gesture recognition system further includes a processor and a memory. The memory stores a gesture recognition program. The gesture recognition program is configured to process The processor is executed to implement the above gesture recognition method.

The present invention also provides a storage medium. The storage medium is a computer-readable storage medium. A gesture recognition program is stored on the storage medium. The gesture recognition program can be executed by one or more processors to realize the above gestures. recognition methods.

Compared with the existing technology, the present invention obtains the gesture feature matrix through two radars at the same time and performs recognition through the convolutional neural network, which can increase the types of gestures recognized and improve the accuracy of gesture recognition.

Description of drawings

Figure 1 is a schematic structural diagram of a gesture recognition system provided by an embodiment of the present invention;

Figure 2 is a flow chart of a gesture recognition method provided by an embodiment of the present invention;

Figure 3 is a schematic sub-flow diagram of step S101 in Figure 1;

Figure 4 is a schematic sub-flow diagram of step S105 in Figure 1;

Figure 5 is a schematic sub-flow diagram of step S107 in Figure 1;

Figure 6 is a comparison chart of recognition accuracy rates of gesture recognition methods provided by embodiments of the present invention.

The realization of the purpose, functional features and advantages of the present invention will be further described with reference to the embodiments and the accompanying drawings.

Detailed ways

In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of the present invention.

The terms "first", "second", "third", "fourth", etc. (if present) in the description and claims of this application and the above-mentioned drawings are used to distinguish similar objects without necessarily using Used to describe a specific order or sequence. It is to be understood that the data so used are interchangeable under appropriate circumstances so that the embodiments described herein can be practiced in sequences other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus that encompasses a series of steps or units and need not be limited to those explicitly listed. Those steps or elements may instead include other steps or elements not expressly listed or inherent to the process, method, product or apparatus.

It should be noted that descriptions involving “first”, “second”, etc. in the present invention are for descriptive purposes only and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of indicated technical features. . Therefore, features defined as "first" and "second" may explicitly or implicitly include at least one of these features. In addition, the technical solutions in various embodiments can be combined with each other, but it must be based on the realization by those of ordinary skill in the art. When the combination of technical solutions is contradictory or cannot be realized, it should be considered that such a combination of technical solutions does not exist. , nor within the protection scope required by the present invention.

An embodiment of the present invention provides a gesture recognition method, which is applied in a gesture recognition system. Please refer to Figure 1. The gesture recognition system includes a first radar 100 and a second radar 200. The gesture recognition system also includes a processor 300 and Memory 400, the gesture recognition system also includes a network interface (not shown) and a communication bus 500. The processor 300 and the memory 400 are connected through a communication bus 500 . The first radar 100 and the second radar 200 are connected through a communication bus 500 . The memory 400 stores a gesture recognition program, the gesture recognition program is configured to be executed by the processor 300, and the processor 300 executes the gesture recognition program to implement the gesture recognition method. Please refer to Figure 2. The gesture recognition method includes:

Step S100: Obtain radar data of the first radar and the second radar within the preset range;

Step S101: Determine whether there is a gesture action in the radar data; if it is determined that there is a gesture action within the preset range, execute step S103; otherwise, return to step S101;

Step S103: Obtain the first gesture information and the second gesture information;

Step S105: Calculate and obtain a gesture feature matrix according to the first gesture information and the second gesture information;

Step S107: Obtain gesture feature values according to the gesture feature matrix and convolutional neural network;

Step S109: Recognize the gesture according to the gesture characteristic value.

In this embodiment, the first gesture information and the second gesture information respectively correspond to the gesture information collected by the first radar and the second radar; the gesture feature matrix includes the first gesture information and the second gesture information. The first characteristic matrix and the second characteristic matrix corresponding to the potential information. Specifically, the transmitting ends of the first radar 100 and the second radar 200 are aligned with the preset gesture range, and the reflected sound waves are received through the receiving ends of the first radar 100 and the second radar 200 . The gesture range is the best range for capturing gesture movements within the radar radiation range. In this embodiment, the signal collected by the radar can be expressed as R _d (m, n _i ). Specifically, the signal collected by the first radar 100 can be expressed as R ₁ (m, n _i ), and the signal collected by the second radar 200 can be expressed as R 1 (m, n i ). The signal is represented by R ₂ (m, _ni ). Among them, n _i is the slow-speed sampling at the i-th moment, and m is the fast-time sampling at the n _i- th pulse.

Preferably, the radar receiving signal contains energy leakage caused by the radar antenna. In order to eliminate the impact on the antenna, before step S101, the gesture recognition method further includes: removing the interference from the receiving end of the first radar 100 or the second radar 200. signal; specifically, collect the gesture-free signal at the second preset time; calculate the self-interference signal according to the gesture-free signal; and remove the self-interference signal received by the first radar or the second radar. It can be understood that by collecting signals without gestures for a period of time, and calculating the stable value of the signals without gestures by averaging, the self-interference signal can be calculated. The signal X _d (m, n _i ) can be obtained by subtracting its own interference signal from the signal collected by the radar. specific,

Remove external interference signals.

Preferably, before step S101, during the gesture detection process, many dynamic interference signals will be generated. This signal may be affected by micro-movements of other parts of the human body or other reasons. These signals can be called Clutter M _d (m, n _i ), based on the linear phase FIR filtering method or the adaptive average clutter subtraction method, suppresses external interference signals. The signal after clutter suppression is F _d (m, _ni ).

Preferably, in order to confirm the timing of starting to recognize the gesture, it is necessary to detect whether there is a gesture within the radar range. For details, please refer to Figure 3. Step S101 includes:

Step S201: Detect whether there is a gesture action based on the radar data; if there is a gesture action, execute step S203; otherwise, return to step S201;

Step S203: Establish plane coordinates with the first radar as the origin;

Step S205: Based on the plane coordinate information, determine whether the gesture action is within the gesture range; if it is determined that the gesture action is within the gesture range, execute step S207; otherwise, return to step S201;

Step S207: Obtain first gesture information and second gesture information according to the radar data.

Specifically, the constant false alarm detector set in the radar can be used to detect whether there is a gesture target, and one of the radars can be used to detect whether the gesture is within the effective range. In some embodiments, if the gesture is not within the gesture range, a reminder may be issued to improve the recognition effect.

Preferably, in step S103, the first gesture information and the second gesture information obtained are respectively the gesture information collected by the first radar 100 and the second radar 200 within the first preset time; in this embodiment , the first preset time is 2 seconds (s).

Preferably, please refer to Figure 4, step S105 includes:

Step S301: Construct a first matrix and a second matrix respectively according to the first gesture information and the second gesture information;

Step S303: Remove outliers in the first matrix and the second matrix;

Step S305: Perform matrix flipping on the first matrix and the second matrix;

Step S307: Complete the first matrix and the second matrix after the matrix flipping, and obtain the first eigenvalue and the second eigenmatrix.

Specifically, the constructed first matrix T ₁ is:

And the second matrix T ₂ is;

In order to avoid outliers in the constructed matrix, it is necessary to remove outliers in the matrix. Specifically, according to the sliding window algorithm, smoothing is performed on the matrix to determine the value of each element on the matrix row. After matrix flipping of the matrix, the first matrix and the second matrix are filled in by zero filling. In this embodiment, the first matrix and the second matrix are completed to a matrix of size 128×128 to obtain the first characteristic matrix and the second characteristic matrix.

Preferably, please refer to Figure 5, the step S107 includes:

Step S401: Convolve and expand the first feature matrix and the second feature matrix respectively to obtain the first convolution and the second convolution;

Step S403: Extend and connect the first convolution and the second convolution into one-dimensional gesture feature values;

Step S405: Obtain gesture features according to the one-dimensional gesture feature value.

In this embodiment, the first feature matrix and the second feature matrix are subjected to three convolution processes respectively. Specifically, the first convolution process first performs 16 filters with a size of 3*3 and a step size of 2. Implement convolution, then perform a 2*2 max pooling layer with a stride of 2 and a linear rectification activation convolution. The second convolution process consists of 32 filters with a size of 3*3 and a stride of 1 and 64 filters with a size of 3*3 and a stride of 1 to implement convolution, and then performs 2*2 with a stride of 1 Convolution with max pooling layer of 2 and linear rectifier activation. The third convolution process is the same as the second convolution process. The first feature matrix and the second feature matrix are expanded by three convolution processes respectively, and the one-dimensional first convolution matrix and the second convolution matrix are respectively output as 8*8*64, and the two matrices are expanded and connected into 8192* The one-dimensional gesture feature value of 1 is input into a normalized exponential function with an output of 14 to identify the gesture action.

Please refer to FIG. 6 , which is a schematic diagram illustrating the comparison of recognition accuracy between the gesture recognition method according to the embodiment of the present invention and the single-radar recognition method 1 and the single-radar recognition method 2 during the same adaptive learning process. The abscissa in Figure 6 represents the percentage of the training set, and the ordinate represents the recognition accuracy. It can be seen from the figure that even when 30% of the training set is used in the embodiment of the present invention, the recognition rate can still reach 96%, and as the training set accounts for When the ratio increases, the test accuracy also increases. When the training set reaches 70%, the test accuracy reaches about 98%. The test accuracy of using one radar alone is much lower than the test accuracy of using two radars.

In this embodiment, two radars can be used to measure two-dimensional coordinates and detect effective gestures, which can reduce the difficulty and interference of subsequent radar gesture recognition; the dual radar system can be used to receive radar gesture signals reflected from different angles, which has The spatial angle and distance information is conducive to the effective recognition of trajectory-type gestures. Through the convolutional neural network, the angle-distance information of the bi-station radar can be adaptively learned. The present invention can be applied in the field of human-computer interaction and control to complete human-computer interaction in places with poor light and harsh environment, such as controlling media playback in the car while driving at night. It can also be used in various control systems in smart homes; it can also be used It is used for effective interaction and control in environments with obstacles, and has important application value for indoor automation applications.

In this embodiment, the memory 400 includes at least one type of readable storage medium. The readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, magnetic disk, optical disk. wait. In some embodiments, the memory 400 may be an internal storage unit of the gesture recognition system, such as a hard disk of the gesture recognition system. In other embodiments, the memory 400 may also be an external storage device of the gesture recognition system, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), or a secure digital (SecureDigital, SD) card equipped on the gesture recognition system. , Flash Card, etc. Further, the memory 400 may also include both an internal storage unit of the gesture recognition system and an external storage device. The memory 400 can not only be used to store application software installed in the gesture recognition system and various types of data, such as codes of the gesture recognition system, etc., but can also be used to temporarily store data that has been output or will be output.

In some embodiments, the processor 300 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chips for running program codes or processes stored in the memory 400 Data, such as executing gesture recognition programs, etc.

The network interface may optionally include a standard wired interface or a wireless interface (such as a WI-FI interface), which is usually used to establish a communication connection between the gesture recognition system and other electronic devices.

Network interfaces are used to implement connection communication between these components.

The communication bus 500 is used to implement connection communication between these components.

In addition, embodiments of the present invention also provide a storage medium on which a gesture recognition program is stored. The gesture recognition program can be executed by one or more processors to achieve the following operations:

Step S101: Determine whether there is a gesture action within the preset range; if it is determined that there is a gesture action within the preset range, execute step S103; otherwise, return to step S101;

Step S103: Obtain the first gesture information and the second gesture information;

Step S105: Calculate and obtain a gesture feature matrix according to the first gesture information and the second gesture information;

Step S107: Obtain the gesture feature value of the feature information according to various matrices of the gesture and the convolutional neural network;

Step S109: Recognize the gesture according to the gesture characteristic value.

The specific implementation of the storage medium of the present invention is basically the same as the above-mentioned gesture recognition method and gesture recognition embodiments, and will not be described again here.

It should be noted that the above serial numbers of the embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. and the terms "comprises," "comprises" or any other variations thereof herein are intended to cover a non-exclusive inclusion such that a process, apparatus, article or method that includes a list of elements includes not only those elements, but also includes not expressly other elements listed, or may also include elements inherent to the process, apparatus, article or method. Without further limitation, an element defined by the statement "comprises a..." does not exclude the presence of other identical elements in a process, device, article or method including that element.

Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus the necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is better. implementation. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product that is essentially or contributes to the existing technology. The computer software product is stored in a storage medium (such as ROM/RAM) as mentioned above. , magnetic disk, optical disk), including a number of instructions for causing a gesture recognition device to perform the methods described in various embodiments of the present invention.

The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transferred from a website, computer, server, or data center Transmission to another website, computer, server or data center by wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store, or a data storage device such as a server or data center integrated with one or more available media. The available media may be magnetic media (eg, floppy disk, hard disk, tape), optical media (eg, DVD), or semiconductor media (eg, Solid State Disk (SSD)), etc.

Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be described again here.

The above are only preferred embodiments of the present invention, and do not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the description and drawings of the present invention may be directly or indirectly used in other related technical fields. , are all similarly included in the scope of patent protection of the present invention.

Claims (6)

1. A gesture recognition method applied to a gesture recognition system including a first radar and a second radar, the method comprising:

judging whether gesture actions exist in the acquired radar data; the radar data includes radar data of a first radar and a second radar;

if the gesture motion exists in the preset range, acquiring first gesture information and second gesture information; the first gesture information and the second gesture information correspond to gesture information acquired by the first radar and the second radar respectively;

according to the first gesture information and the second gesture information, a gesture feature matrix is obtained; the gesture feature matrix comprises a first feature matrix and a second feature matrix corresponding to the first gesture information and the second gesture information;

acquiring a gesture characteristic value according to the gesture characteristic matrix and the convolutional neural network;

recognizing a gesture according to the gesture characteristic value;

the obtaining the gesture feature matrix according to the first gesture information and the second gesture information includes:

respectively constructing a first matrix and a second matrix according to the first gesture information and the second gesture information;

removing abnormal values in the first matrix and the second matrix;

the first matrix and the second matrix are subjected to matrix overturning, and the first matrix and the second matrix after matrix overturning are completed through a zero filling method, so that a first characteristic matrix and a second characteristic matrix are obtained;

before judging whether gesture actions exist in the preset range, the gesture recognition method further comprises the following steps:

calculating a stable value of the signal without gestures by a mean value calculating method, and calculating to obtain an interference signal;

removing self interference signals of the first radar or the second radar receiving end;

removing external interference signals;

"determining whether a gesture exists in the radar data" includes:

detecting whether gesture actions exist according to the radar data;

if gesture action exists, establishing plane coordinates by taking the first radar as an origin;

judging whether the gesture is in a gesture range according to the plane coordinate information;

if the gesture is judged to be in the gesture range, acquiring first gesture information and second gesture information according to the radar data; the gesture range is a common detection range preset by the first radar and the second radar; if the gesture is judged not to be in the gesture range, a prompt is sent out;

the step of acquiring the gesture feature value according to the gesture feature matrix and the convolutional neural network comprises the following steps:

performing convolution expansion on the first feature matrix and the second feature matrix respectively to obtain a first convolution and a second convolution;

connecting the first convolution and the second convolution expansion into a one-dimensional gesture characteristic value;

recognizing gestures according to the one-dimensional gesture characteristic values;

the first characteristic matrix and the second characteristic matrix are respectively subjected to three convolution processes, the first convolution process is performed with 16 filters with the size of 3*3 and the step length of 2 to realize convolution, and then the maximum pooling layer with the step length of 2 and the linear rectification activation convolution are performed with 2 x 2; the second convolution processing is carried out by using 32 filters with the size of 3*3 and the step size of 1 and 64 filters with the size of 3*3, the step size of 1 to realize convolution, and then carrying out 2 x 2, the maximum pooling layer with the step size of 2 and linear rectification to activate convolution; the third convolution processing is the same as the second convolution processing; the first characteristic matrix and the second characteristic matrix are respectively subjected to three convolution processing expansion, 8-by-64 one-dimensional first convolution matrix and second convolution matrix are respectively output, the two matrices are expanded and connected into 8192-by-1 one-dimensional gesture characteristic values, the one-dimensional gesture characteristic values are input into a 14-output normalized exponential function, and the gesture motion is identified.

2. The gesture recognition method of claim 1, wherein: the "removing outliers in the first matrix and the second matrix" includes:

and carrying out smoothing processing on the data of the first matrix and the second matrix according to a sliding window algorithm.

3. The gesture recognition method of claim 1, wherein: the "removing the self-interference signal of the first radar or the second radar receiving end" includes:

acquiring gesture-free signals of a second preset time;

according to the gesture-free signal, calculating an interference signal;

and removing the self interference signal received by the first radar or the second radar.

4. A gesture recognition method according to claim 3, wherein: the step of removing the external interference signal specifically comprises the step of removing the external interference signal according to a clutter suppression method; the clutter suppression method comprises the following steps: linear phase FIR filtering, adaptive average clutter subtraction and adaptive iterative clutter subtraction.

5. A gesture recognition system comprising a first radar and a second radar, characterized in that: the gesture recognition system further comprises a processor and a memory, the memory storing a gesture recognition program configured to be executed by the processor to implement the gesture recognition method of any one of claims 1-4.

6. A storage medium, characterized by: the storage medium is a computer readable storage medium having stored thereon a gesture recognition program executable by one or more processors to implement the gesture recognition method of any one of claims 1-4.