WO2018137456A1 - Visual tracking method and device - Google Patents

Abstract

A visual tracking method and a visual tracking device, for use in resolving the technical problem of inability to accurately position an eye object in real time. The method comprises: obtaining an eye pattern (10); building an eye object key point (20); processing the eye pattern by using the eye object key point as a test data of an object processing method, and determining an eye object position to form visual focus data (30). Processing of a large number of redundancy image signals is avoided, and the calculation amount of image processing is decreased. To build the eye object key point, supervision or semi-supervision learning is used, a quantization tool is utilized to form high calibration data quality, key point calibration data achieves a directional cutting effect on classification of eye objects, such as iris data, in the image processing method, and accurate positioning of the eye objects is met. Moreover, the present invention facilitates further determining other eye objects, such as pupil boundary, and further forming accurate visual focus and visual motion trajectory.

Description

Visual tracking method and tracking device

The present invention claims priority from the applicant's application with the filing date of January 25, 2017, application number CN201710060901.3, entitled "A Visual Tracking Method and Tracking Device". The entire contents of the above application are hereby incorporated by reference in its entirety.

Technical field

Embodiments of the present invention relate to the field of bio-feedback signal data processing technologies, and in particular, to a visual signal tracking method and a tracking device.

Background of the invention

Visual recognition and tracking are mainly to judge the attention direction and attention trajectory of the pupil of the eye. Since the pupil is composed of physiological organs and tissues such as the sclera and the iris, the pupils have a large individualized difference, and the difference of the iris is the largest. The current technical means mainly adopts image recognition for the eye, and uses the binary feature, the gradient histogram, etc., combined with the filtering operation such as expansion corrosion to achieve the purpose of extracting the iris position. However, the above method is basically based on prior knowledge. For complex biological individual differences, it is necessary to determine more hypothetical parameter sets and threshold ranges, which can only be effective in a limited scene, and the accuracy is low, and the iris of the dynamic image cannot be targeted for real-time scenes. deal with.

Summary of the invention

In view of this, the embodiments of the present invention provide a visual tracking method and a tracking device, which are used to solve the technical problem that an eye object cannot be accurately located in real time.

The visual tracking method of the embodiment of the invention includes:

Obtaining an eye pattern;

Establish key points of the eye object;

The eye pattern is processed using the eye object key point as the test data of the object processing method, and the eye object position is determined to form visual focus data.

Also includes:

Continuously changing the eye object to form visual tracking data;

The visual tracking data is used as a control signal for the action change of the virtual vision.

The acquiring an eye pattern includes:

Get the facial features of the face;

A symmetrical eye image is cropped according to the eye feature points.

The key points for establishing an eye object include:

Establishing an eye object in a semi-manual or automated manner;

The key points of the eye object are formed in a semi-manual or automated manner.

The processing the eye pattern by using the eye object key point as the test data of the object processing method, determining the position of the eye object, and forming the visual focus data includes:

Importing the pixel data of the eye pattern into the ERT algorithm as training data for processing;

Determining the determined eye object and the eye object key point as test data to correct the processing result of the ERT algorithm;

According to the corrected processing result, an accurate contour of the eye object and an accurate relative positional relationship are formed.

The continuous change of the eye object to form visual tracking data includes:

In the eye pattern acquired in real time, visual tracking data is formed according to the relative position change of the eye object;

In the eye pattern acquired in real time, visual tracking data is formed according to the relative positional change of the corresponding key point of the eye object.

The action changes for using the visual tracking data as a control signal for virtual vision include:

Establishing key points of the eye object and/or the eye object, and mapping with the key points of the eye 3D model or the 2D model and the object to form a virtual visual focus;

The visual tracking data is used to control the movement of key points in the three-dimensional model or the two-dimensional model of the eye and/or the object to form a change in the virtual vision.

The visual tracking device of the embodiment of the invention includes:

An image acquisition module, configured to acquire an eye pattern;

Key data creation module for establishing key points of eye objects;

The object recognition module is configured to process the eye pattern by using the eye object key point as the test data of the object processing method, determine the iris position, and form the visual focus data.

Also includes:

a visual tracking data generating module, configured to continuously change the eye object to form visual tracking data;

A virtual vision control module for using visual tracking data as a control signal for motion changes of virtual vision.

The image acquisition module includes:

a contour acquisition sub-module for acquiring a facial features of the face;

An image cropping sub-module for cropping a symmetrical eye image according to an eye feature point.

The key data establishing module includes:

An eye object creation sub-module for establishing an eye object in a semi-manual or automatic manner;

The object key points establish sub-modules for forming key points of the eye object in a semi-manual or automatic manner.

The object recognition module includes:

An image importing sub-module, configured to import pixel data of an eye pattern into an ERT algorithm for processing as a training data;

An image processing sub-module, configured to use the determined eye object and the eye object key point as test data to correct the processing result of the ERT algorithm;

The eye object position generation sub-module is configured to form an accurate contour of the eye object and an accurate relative positional relationship according to the corrected processing result.

The visual tracking data generating module includes:

An eye object trajectory generating sub-module, configured to form visual tracking data according to a relative position change of the eye object in the eye pattern acquired in real time;

The object key point trajectory generating sub-module is configured to form visual tracking data according to a relative position change of a corresponding key point of the eye object in the eye pattern acquired in real time.

The virtual vision control module includes:

a virtual focus generation sub-module for establishing a key point of an eye object and/or an eye object, and mapping with an eye object in a three-dimensional model or a two-dimensional model and a key point of the object to form a virtual visual focus;

The virtual vision generation sub-module is configured to control the movement of the key points of the three-dimensional model or the two-dimensional model of the eye and/or the object by using the visual tracking data to form a change of the virtual vision.

A visual tracking device according to an embodiment of the present invention includes a processor and a memory,

The memory is used in the program code of the visual tracking method described above;

The processor is configured to execute the program code.

The visual tracking method and the visual tracking device of the embodiments of the present invention determine the eye pattern based on the mature face detection technology, avoid processing a large number of redundant image signals, and simplify the calculation amount of the image processing. The key points of the eye are established by supervised or semi-supervised learning, using the quantitative tools to form a high quality of the calibration data, and the key point calibration data in the image processing method has the effect of directional cutting of the classification of the eye objects such as iris data. Meet the exact positioning of the eye object. It is also useful to further determine other eye objects such as pupil boundaries, and further form accurate visual focus and visual motion trajectory.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart of a visual tracking method according to an embodiment of the present invention.

2 is a flow chart of a visual tracking method according to an embodiment of the present invention.

FIG. 3 is a flowchart of a visual tracking method according to an embodiment of the present invention.

4 is a schematic diagram of 68 feature points of a facial facial features determined in the prior art.

FIG. 5 is a schematic structural diagram of key points of an eye object in a left eye pattern in a visual tracking method according to an embodiment of the present invention.

FIG. 6 is a schematic structural diagram of a visual tracking device according to an embodiment of the present invention.

FIG. 7 is a schematic structural diagram of a visual tracking apparatus according to an embodiment of the present invention.

FIG. 8 is a schematic structural diagram of a visual tracking apparatus according to an embodiment of the present invention.

Mode for carrying out the invention

The technical solutions in the embodiments of the present invention are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present invention. It is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts are within the scope of the present invention.

The step numbers in the drawings are only used as reference numerals for the steps, and do not indicate the order of execution.

FIG. 1 is a flow chart of a visual tracking method according to an embodiment of the present invention. As shown in FIG. 1, the visual tracking method of the embodiment of the present invention includes:

Step 10: Obtain an eye pattern;

Step 20: Establish a key point of the eye object;

Step 30: The eye pattern is processed by using the eye object key point as test data of the object processing method, and the eye object position is determined to form visual focus data.

The visual tracking method of the embodiment of the present invention determines the eye pattern based on the mature face detection technology, avoids the processing of a large number of redundant image signals, and simplifies the calculation amount of the image processing. Eye key points are established using supervised or semi-supervised learning, using quantitative tools to form a high quality of calibration data, and key point calibration data is used to classify eye objects such as iris data in, for example, ERT (Ensemble of Regression Trees) processing methods. It has the effect of directional cropping, which satisfies the accurate positioning of the iris boundary of the eye object. It is also useful to further determine other eye objects such as pupil boundaries.

2 is a flow chart of a visual tracking method according to an embodiment of the present invention. As shown in FIG. 2, based on the foregoing embodiment, the visual tracking method of the embodiment of the present invention further includes:

Step 40: Continuously change the eye object to form visual tracking data;

Step 50: Use the visual tracking data as a control signal for the action change of the virtual vision.

The visual tracking method of the embodiment of the invention forms the visual tracking data of the continuous visual focus data of the human eye, and based on the mature coordinate change process, the corresponding actions of the iris object and the pupil object of the eye of the anthropomorphic object can be formed, and the anthropomorphic object pair is realized. The synchronous positive feedback of the human visual eye enriches the emotional expression of the anthropomorphic object.

FIG. 3 is a flowchart of a visual tracking method according to an embodiment of the present invention. As shown in FIG. 3, in the visual tracking method of an embodiment of the present invention, step 10 further includes:

Step 11: Obtain the facial features of the face.

The acquisition of the facial features, such as the dlib face detection model, yields 68 feature points (as shown in Figure 4). It is important to be clear that the feature points of the facial features cannot be used to accurately describe the position and characteristics of the facial features.

Step 12: Crop a symmetrical eye image according to the eye feature points.

As shown in FIG. 4 and FIG. 5, the cropping takes 68 feature points as an example, and the left eye pattern surrounded by the eye points of the feature points 37-42 and the right eye pattern surrounded by the feature points 43-48 are respectively cut by the minimum circumscribed rectangle algorithm.

In the visual tracking method of an embodiment of the invention, step 20 further includes:

Step 21: Establish an eye object in a semi-manual or automatic manner.

For the semi-manual method: including the artificial marker to determine the approximate range of the eye object, based on the artificial marker, the image recognition algorithm is used to further determine the approximate range of the eye object.

For automated methods, including the use of image recognition algorithms to determine the approximate range of eye objects. The image recognition algorithm establishes an approximate range of eye objects by a mapping pattern on a two-dimensional plane during the movement of the three-dimensional model of the established eye object.

Step 22: Form a key point of the eye object in a semi-manual or automatic manner.

For the semi-manual method: in the range of the determined eye object, the key points are manually marked, and on the basis of the artificial mark, the image recognition algorithm is used to further mark the key points of the eye object concealment.

For automatic methods, including the use of image recognition algorithms to determine the key points of the eye object. The image recognition algorithm marks the key points of the eye object through the mapping points on the two-dimensional plane during the movement of the three-dimensional model of the established eye object.

The manual and automatic combination of a small number of eye images can effectively improve the processing speed and ensure the accuracy, which provides a guarantee for further ensuring the accuracy of the subsequent algorithms as training data. A large number of eye images are automatically processed to ensure the speed of dynamic vision processing.

In the visual tracking method of an embodiment of the present invention, the eye object determined in step 20 includes:

The 12 key points of the eyelids and eyelids include the key points at the ends of the eyelids and the key points at the maximum distance between the upper eyelid and the lower eyelid.

The eight key points of the iris and the iris include the key points at the maximum distance between the left and right edges of the iris in the horizontal direction and the maximum points at the maximum distance between the upper and lower edges of the iris in the vertical direction.

The eight key points of the pupil and the pupil include the key point at the maximum distance between the left edge and the right edge of the pupil in the horizontal direction and the key point at the maximum distance between the upper edge and the lower edge of the pupil in the vertical direction.

Key points include the corresponding coordinate position and pattern properties.

In the visual tracking method of an embodiment of the invention, step 30 includes:

Step 31: Import pixel data of the eye pattern into the ERT algorithm as training data for processing;

Step 32: The determined eye object and the eye object key point are used as test data to correct the processing result of the ERT algorithm;

Step 33: Form an accurate contour of the eye object and an accurate relative positional relationship according to the corrected processing result.

The key point data obtained by manual or semi-manual processing is used as test data to ensure the prediction accuracy in the process of the ERT algorithm.

In the visual tracking method of an embodiment of the invention, the step 40 further includes:

Step 41: forming visual tracking data according to a relative position change of the eye object in the eye pattern acquired in real time;

Step 42: In the eye pattern acquired in real time, visual tracking data is formed according to the relative position change of the corresponding key point of the eye object.

The visual tracking method of the embodiment of the present invention has two significant advantages after practical application and comparative measurement:

1, the accuracy is high, the iris position error does not exceed 3% (refers to the actual iris position and predicted iris position distance in addition to the maximum distance of the lower eyelid);

2, robustness and real-time performance, the determination of eye objects on ordinary computers and mobile devices does not exceed 3ms / frame, calculated at 30 frames / second, the recognition failure rate is less than 0.5%.

In the visual tracking method of an embodiment of the invention, step 50 further includes:

Step 51: Establish a key point of the eye object and/or the eye object, and map the key points of the object and the object in the three-dimensional model or the two-dimensional model of the eye to form a virtual visual focus;

Step 52: Control the movement of the key points of the object and/or the object in the three-dimensional model or the two-dimensional model of the eye using the visual tracking data to form a change of the virtual vision.

The visual tracking method of the embodiment of the invention can apply the obtained visual tracking data to the eye expression of the virtual object, thereby further improving the anthropomorphic feature of the virtual object.

FIG. 6 is a schematic structural diagram of a visual tracking device according to an embodiment of the present invention. As shown in FIG. 6, the visual tracking method corresponding to the embodiment of the present invention further includes a visual tracking device, including:

The image acquisition module 100 is configured to acquire an eye pattern.

The key data creation module 200 is configured to establish an eye object key point.

The object recognition module 300 is configured to process the eye pattern by using the eye object key point as the test data of the object processing method, determine the iris position, and form the visual focus data.

FIG. 7 is a schematic structural diagram of a visual tracking apparatus according to an embodiment of the present invention. As shown in FIG. 7, the visual tracking device according to an embodiment of the present invention further includes:

The visual tracking data generating module 400 is configured to form a visual tracking data by continuously changing the eye object.

The virtual vision control module 500 is configured to use the visual tracking data as a control signal for the motion change of the virtual vision.

FIG. 8 is a schematic structural diagram of a visual tracking apparatus according to an embodiment of the present invention. As shown in FIG. 8, in the visual tracking device of an embodiment of the present invention, the image acquisition module 100 includes:

The contour acquisition sub-module 110 is configured to acquire a facial features of the face.

The image cropping sub-module 120 is configured to crop a symmetrical eye image according to the eye feature points.

In the visual tracking device of an embodiment of the present invention, the key data establishing module 200 includes:

The eye object creation sub-module 210 is configured to establish an eye object in a semi-manual or automatic manner.

The object key point sub-module 220 is configured to form a key point of the eye object in a semi-manual or automatic manner.

In the visual tracking device of an embodiment of the invention, the object recognition module 300 includes:

The image importing sub-module 310 is configured to import the pixel data of the eye pattern into the ERT algorithm for processing as the training data;

The image processing sub-module 320 is configured to correct the determined eye object and the eye object key point as test data to correct the processing result of the ERT algorithm;

The eye object position generation sub-module 330 is configured to form an accurate contour of the eye object and an accurate relative positional relationship according to the corrected processing result.

In the visual tracking device of an embodiment of the present invention, the visual tracking data generating module 400 includes:

The eye object trajectory generating sub-module 410 is configured to form visual tracking data according to a relative position change of the eye object in the eye pattern acquired in real time;

The object key point trajectory generation sub-module 420 is configured to form visual tracking data according to a relative position change of a corresponding key point of the eye object in the eye pattern acquired in real time.

In the visual tracking device of an embodiment of the invention, the virtual visual control module 500 includes:

The virtual focus generation sub-module 510 is configured to establish a key point of the eye object and/or the eye object, and map the key points of the object and the object in the three-dimensional model or the two-dimensional model of the eye to form a virtual visual focus;

The virtual vision generation sub-module 520 is configured to control the movement of the key points in the three-dimensional model or the two-dimensional model of the eye and/or the key points of the object by using the visual tracking data to form a change of the virtual vision.

The above is only the preferred embodiment of the present invention, and is not intended to limit the present invention. Any modifications, equivalent substitutions, etc., which are within the spirit and principles of the present invention, should be included in the scope of the present invention. within.

A visual tracking device in accordance with an embodiment of the present invention includes a memory and a processor, wherein:

a memory for storing program code for implementing the processing steps of the visual tracking method of the above embodiment;

The processor is for executing program code that implements the processing steps of the visual tracking method of the above-described embodiments.

Those of ordinary skill in the art will appreciate that the elements and algorithm steps of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the solution. A person skilled in the art can use different methods for implementing the described functions for each particular application, but such implementation should not be considered to be beyond the scope of the present invention.

A person skilled in the art can clearly understand that for the convenience and brevity of the description, the specific working process of the system, the device and the unit described above can refer to the corresponding process in the foregoing method embodiment, and details are not described herein again.

In the several embodiments provided herein, it should be understood that the disclosed systems, devices, and methods may be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the unit is only a logical function division. In actual implementation, there may be another division manner, for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in an electrical, mechanical or other form.

The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

Functionality, if implemented as a software functional unit and sold or used as a stand-alone product, can be stored on a computer readable storage medium. Based on such understanding, the technical solution of the present invention, which is essential or contributes to the prior art, or a part of the technical solution, may be embodied in the form of a software product, which is stored in a storage medium, including The instructions are used to cause a computer device (which may be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and the like, and can store a program check code. Medium.

The above is only a specific embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can easily think of changes or substitutions within the technical scope of the present invention. It should be covered by the scope of the present invention. Therefore, the scope of the invention should be determined by the scope of the claims.

Industrial applicability

The visual tracking method and the tracking device of the embodiment of the present invention determine the eye pattern, avoid processing of a large number of redundant image signals, and simplify the calculation amount of image processing. Using the quantification tool to form a high quality of the calibration data, the key point calibration data has the effect of directional clipping on the classification of eye objects such as iris data, and satisfies the accurate positioning of the iris boundary of the eye object. It is also useful to further determine other eye objects such as pupil boundaries. The visual tracking method and the tracking device of the embodiments of the present invention determine that the eye pattern can be generally applied to the smart mobile terminal device, improving the efficiency of the human-computer interaction process.

Claims (15)

A visual tracking method comprising: Obtaining an eye pattern; Establish key points of the eye object; The eye pattern is processed using the eye object key point as the test data of the object processing method, and the eye object position is determined to form visual focus data. The visual tracking method of claim 1 further comprising: Continuously changing the eye object to form visual tracking data; The visual tracking data is used as a control signal for the action change of the virtual vision. The visual tracking method according to claim 2, wherein the acquiring the eye pattern comprises: Get the facial features of the face; A symmetrical eye image is cropped according to the eye feature points. The visual tracking method according to claim 2, wherein the establishing an eye object key point comprises: Establishing an eye object in a semi-manual or automated manner; The key points of the eye object are formed in a semi-manual or automated manner. The visual tracking method according to claim 2, wherein the processing the eye pattern by using the eye object key point as the test data of the object processing method, determining the position of the eye object, and forming the visual focus data comprises: Importing the pixel data of the eye pattern into the ERT algorithm as training data for processing; Determining the determined eye object and the eye object key point as test data to correct the processing result of the ERT algorithm; According to the corrected processing result, an accurate contour of the eye object and an accurate relative positional relationship are formed. The visual tracking method according to claim 2, wherein the continuously changing the eye object to form visual tracking data comprises: In the eye pattern acquired in real time, visual tracking data is formed according to the relative position change of the eye object; In the eye pattern acquired in real time, visual tracking data is formed according to the relative positional change of the corresponding key point of the eye object. The visual tracking method according to claim 2, wherein the action change for using the visual tracking data as a control signal for virtual vision comprises: Establishing key points of the eye object and/or the eye object, and mapping with the key points of the eye 3D model or the 2D model and the object to form a virtual visual focus; The visual tracking data is used to control the movement of key points in the three-dimensional model or the two-dimensional model of the eye and/or the object to form a change in the virtual vision. A visual tracking device comprising: An image acquisition module, configured to acquire an eye pattern; Key data creation module for establishing key points of eye objects; The object recognition module is configured to process the eye pattern by using the eye object key point as the test data of the object processing method, determine the iris position, and form the visual focus data. The visual tracking device of claim 8 further comprising: a visual tracking data generating module, configured to continuously change the eye object to form visual tracking data; A virtual vision control module for using visual tracking data as a control signal for motion changes of virtual vision. The visual tracking device of claim 9, wherein the image acquisition module comprises: a contour acquisition sub-module for acquiring a facial features of the face; An image cropping sub-module for cropping a symmetrical eye image according to an eye feature point. The visual tracking device of claim 9, wherein the key data establishing module comprises: An eye object creation sub-module for establishing an eye object in a semi-manual or automatic manner; The object key points establish sub-modules for forming key points of the eye object in a semi-manual or automatic manner. The visual tracking device of claim 9, wherein the object recognition module comprises: An image importing sub-module, configured to import pixel data of an eye pattern into an ERT algorithm for processing as a training data; An image processing sub-module, configured to use the determined eye object and the eye object key point as test data to correct the processing result of the ERT algorithm; The eye object position generation sub-module is configured to form an accurate contour of the eye object and an accurate relative positional relationship according to the corrected processing result. The visual tracking device of claim 9, wherein the visual tracking data generating module comprises: An eye object trajectory generating sub-module, configured to form visual tracking data according to a relative position change of the eye object in the eye pattern acquired in real time; The object key point trajectory generating sub-module is configured to form visual tracking data according to a relative position change of a corresponding key point of the eye object in the eye pattern acquired in real time. The visual tracking device of claim 9, wherein the virtual visual control module comprises: a virtual focus generation sub-module for establishing a key point of an eye object and/or an eye object, and mapping with an eye object or a key point of the object in the two-dimensional model to form a virtual visual focus; The virtual vision generation sub-module is configured to control the movement of the key points of the three-dimensional model or the two-dimensional model of the eye and/or the object by using the visual tracking data to form a change of the virtual vision. A visual tracking device includes a processor and a memory, wherein The memory is configured to store program code for completing the visual tracking method of any one of claims 1 to 7; The processor is configured to execute the program code.

Images