CN117830783B - Sight estimation method based on local super-resolution fusion attention mechanism - Google Patents

Abstract

The present invention belongs to the field of computer vision, and specifically relates to a line of sight estimation method based on a local super-resolution fusion attention mechanism. The method comprises the following steps: step S1, using a camera to obtain a frame image; step S2, using a face detection model to detect and locate the face area and the binocular area, cropping the face image, and intercepting the eye image; step S3, passing the face image through a face attention enhancement feature extraction module to enhance and extract face image features; step S4, passing the binocular image through an eye feature extraction module based on local super-resolution to extract binocular image features; step S5, fusing the extracted face image and binocular image features through a fully connected layer to obtain a line of sight estimation result. The present invention extracts the eye features after super-resolution, performs accurate line of sight estimation, and enhances the low-resolution global features from both spatial and channel directions to increase the ability to extract face features in a low-resolution environment and improve the effect of line of sight estimation.

Description

A gaze estimation method based on local super-resolution fusion attention mechanism

Technical Field

The present invention belongs to the field of computer vision, and in particular relates to a line of sight estimation method based on a local super-resolution fusion attention mechanism.

Background Art

Gaze estimation is a key subfield in computer vision that focuses on determining the direction in which an individual's eyes are looking. Gaze behavior is an important part of human social interaction. By analyzing the direction of gaze, a lot of potential information can be obtained. For example, shopping malls can analyze which products are most popular based on customers' gaze data; invigilators can determine whether students are suspected of cheating based on their gaze direction. In addition, gaze estimation technology has been widely used in many fields, such as virtual reality, driver assistance systems, human-computer interaction, etc., and gaze estimation has broad application prospects in these fields.

With the development of deep learning, convolutional neural network-based line of sight estimation methods have become popular. However, these methods usually require a large amount of data sets and are mainly tested under ideal conditions. Usually, these methods are trained mainly with high-resolution facial images. In fact, factors such as camera quality and facial distance often lead to unclear facial input. The above-mentioned customer line of sight analysis and exam room proctoring cases will cause unclear input images due to factors such as camera support clarity and human distance. The face and eye images at different resolutions are shown in Figure 1. As the resolution of the input image decreases, information will gradually be lost, which will make it increasingly difficult for the network to extract features, resulting in a decrease in the accuracy of line of sight estimation.

In actual scenarios, input images are often affected by many factors. Due to low camera resolution and long face distance, input face images often have low clarity and information is gradually lost, which makes it increasingly difficult for the network to extract features, leading to a decrease in the accuracy of line of sight estimation. The current line of sight estimation technology has a low accuracy rate in low-resolution scenarios, and no effective solution has yet been found.

Summary of the invention

The purpose of the present invention is to overcome the shortcomings of the prior art and propose a line of sight estimation method based on local super-resolution fusion attention mechanism.

The technical solution adopted by the method of the present invention is as follows:

A line of sight estimation method based on local super-resolution fusion attention mechanism includes the following steps:

Step S1, using a camera to obtain a frame image;

Step S2: using a face detection model to detect and locate the face area and eye area, cropping the face image, and capturing the eye image;

Step S3, passing the face image through the face attention enhancement feature extraction module to enhance and extract the face image features;

Step S4, extracting binocular image features from the binocular images through an eye feature extraction module based on local super-resolution;

Step S5: fusing the extracted face image and binocular image features through a fully connected layer to obtain a sight line estimation result.

Further, as a preferred technical solution of the present invention, in step S2, the face detection model adopts a face detection model based on a convolutional neural network. Compared with the traditional face detection model, the face detection model using a convolutional neural network has higher accuracy, can handle various complex situations, and has stronger robustness, real-time performance and adaptability. Specifically, the "dlib.cnn_face_detection_model_v1" model is used, which includes the weights and structure of the convolutional neural network, which are obtained by training on a large amount of image data. After loading this model, faces can be detected on images.

Further, as a preferred technical solution of the present invention, in step S3, the face attention enhancement feature extraction module uses the ResNet18 model as the benchmark model of the face feature extraction module. For each standard residual block, the formula is as follows:

F _out =l(f(F _in ,W _i )+F _in ) (1)

Among them, l represents the ReLU activation function, f is the weight operation in the residual block, W is the weight of the operation, _Fin is the input of the residual block, and _Fout represents the output of the residual block.

Further, as a preferred technical solution of the present invention, in step S3, the face attention enhancement feature extraction module adds a CBAM attention mechanism on the basis of the ResNet18 benchmark model to increase the accuracy of line of sight estimation; the CBAM attention mechanism adds two attention modules to the feature map: a channel attention module and a spatial attention module; the channel attention module aims to assign a weight to each channel, which is generally achieved by the features obtained by global average pooling and global maximum pooling. For a given feature map F∈R ^C×H×W , the calculation first calculates the global average pooling and the global maximum pooling, then the channel attention can be expressed as these two values are processed by a shared multi-layer perceptron and combined, and the formula is as follows:

Where σ represents the Sigmoid activation function and M _CA represents channel attention. represents global average pooling, represents global maximum pooling;

The spatial attention module aims to assign a weight to each space. First, the global average pooling and global maximum pooling are still calculated, but this time along the channel dimension. Finally, the two feature maps are concatenated and passed through a 7×7 convolution layer, and then through a Sigmoid activation function. The formula is as follows:

M _SA (F)=σ(Conv ^7×7 ([F _gap ; F _gmp ])) (3)

Conv ^7×7 represents the 7×7 convolutional layer processing.

The face attention enhanced feature extraction module adds a CBAM attention module at the end of each stage to enhance the features. For each residual stage, the formula can be expressed as follows:

Where ↓ represents downsampling. Then the features enhanced by the CBAM attention mechanism can be expressed as:

in represents the multiplication of corresponding elements, and F _FA represents the facial feature map after CBAM attention enhancement.

Further, as a preferred technical solution of the present invention, in step S4, the eye feature extraction module based on local super-resolution adopts FSRCNN as the network for super-resolution reconstruction. The FSRCNN algorithm is a deep convolutional neural network model designed for single image super-resolution. It is further improved on the basis of SRCNN. The purpose is to improve the speed and efficiency of super-resolution reconstruction.

The FSRCNN algorithm is mainly divided into three parts: feature extraction, contraction and expansion, and deconvolution. The first part is feature extraction, which uses a smaller convolution kernel of size 5×5 to extract features from low-resolution eye images. The purpose is to extract useful information from the original low-resolution image, which will then be used to reconstruct a high-resolution image. The formula for the feature extraction stage is as follows:

Where I _LR represents the low-resolution eye image, PRELU represents the PreLU activation function, and d represents the number of feature maps.

The second part is contraction and expansion. The contraction stage uses a 1×1 convolution kernel to reduce the number of feature maps. The purpose of this stage is to reduce the number of model parameters and thus improve the operation speed. The mapping stage performs nonlinear mapping on the contracted feature space through multiple consecutive 3×3 convolution layers. The formula that can be expressed in the feature mapping stage is as follows:

The third part is deconvolution, which uses the deconvolution layer to expand the spatial size of the feature map to obtain a high-resolution output image. Different from the traditional bicubic interpolation method, the deconvolution at this stage is a learning method. This stage can convert the low-resolution feature map into the feature map of the high-resolution image. The formula is as follows:

F _SR = DeConv ^9×9 (F) (9)

Where DeConv represents the deconvolution operation.

Further, as a preferred technical solution of the present invention, in step S4, the eye feature extraction module based on local super-resolution uses eye feature extraction DeepEyeNet to extract features; DeepEyeNet is a CNN network for deep eye feature extraction proposed in the present invention, which consists of ten convolution blocks. This is a relatively deep convolutional neural network, and the spatial size of the feature map gradually decreases in the depth of the network. This deep network can better extract eye features after super-resolution to accurately estimate the line of sight; both the left and right eyes go through a similar structure, and the formula is as follows:

F′ _E =FC _E (FLAT(γ(F _SR ))) (10)

Where γ represents the feature extraction operation of DeepEyeNet, F′ _E represents the output eye feature, and FC _E represents the eye fully connected operation;

After obtaining the features of the face and eyes, the network passes through a fully connected layer to connect the features of the face, left eye, and right eye, and outputs the final two-dimensional features as the line of sight estimation result.

ξ _pred =FC _T ([FC _FA (F _FA ); F′ _E ]) (11)

Where FC _FA represents the fully connected face operation.

Further, as a preferred technical solution of the present invention, MSELoss is used as the loss function of line of sight estimation. MSELoss is also called mean square error, which represents the expected value of the square of the error between the model estimated prediction value and the true value. The smaller the MSELoss, the larger the error. Therefore, the loss function of the line of sight estimation of this module is:

Among them, the true value of the line of sight is ξ _gt , and the predicted value of the line of sight estimation is ξ _pred .

Compared with the prior art, the present invention has the following beneficial effects:

(1) The present invention designs an eye feature extraction module based on local super-resolution. The network first performs super-resolution reconstruction on the eye image to efficiently and quickly restore the low-resolution line of sight related features. Then, it passes through a novel deep convolutional neural network. The spatial size of the feature map gradually decreases in the depth of the network. This network can better extract the eye features after super-resolution to perform accurate line of sight estimation.

(2) The present invention proposes a face attention enhanced feature extraction network. The present invention improves the ordinary ResNet18 feature extraction network and enhances low-resolution global features from both spatial and channel directions to increase the ability to extract face features in low-resolution environments, thereby improving the effect of line of sight estimation.

(3) The present invention improves the accuracy of line of sight estimation in low-resolution environments.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

FIG1 is a schematic diagram of a face and two-eye image at different resolutions in the prior art;

FIG2 is a flow chart of a method according to an embodiment of the present invention;

FIG3 is a diagram of the overall network framework of an embodiment of the present invention;

FIG4 is a structural diagram of the attention improvement residual stage according to an embodiment of the present invention;

FIG5 is a structural diagram of an eye feature extraction module according to an embodiment of the present invention.

DETAILED DESCRIPTION

The present invention is further explained below in detail with reference to the accompanying drawings so that those skilled in the art can more deeply understand the present invention and be able to implement it. However, the following reference examples are only used to explain the present invention and are not intended to limit the present invention.

Figure 1 shows the face and eye images at different resolutions. It can be seen that in the 128×128 HR scene, the information is relatively rich, and the features of both eyes and the whole face can be completely extracted. We can also distinguish the direction of the person's line of sight with the naked eye. In the 64×64, 32×32, and 16×16 LR scenes, as the resolution gradually decreases, information is gradually lost, and it becomes increasingly difficult to extract features. In the extreme case of 16×16, even the shape of the eyes cannot be distinguished, which undoubtedly increases the difficulty of line of sight estimation.

As shown in Figure 2, a line of sight estimation method based on local super-resolution fusion attention mechanism includes the following steps: step S1, using a camera to obtain a frame image; step S2, using a face detection model to detect and locate the face area and the eye area, cropping the face image, and intercepting the eye image; step S3, passing the face image through a face attention enhancement feature extraction module to enhance and extract face image features; step S4, passing the eye feature extraction module based on local super-resolution to extract the eye image features; step S5, fusing the extracted face image and eye image features through a fully connected layer to obtain a line of sight estimation result.

In step S2, a face detection model based on a convolutional neural network is used. Compared with the traditional face detection model, the face detection model using a convolutional neural network has higher accuracy, can handle various complex situations, and has stronger robustness, real-time performance and adaptability. Specifically, the "dlib.cnn_face_detection_model_v1" model is used, which contains the weights and structure of the convolutional neural network, which are obtained by training on a large amount of image data. After loading this model, faces can be detected on images.

In step S3, the ResNet18 model is used as the benchmark model of the facial feature extraction module.

The residual neural network was first proposed by He et al. The core part of this network structure is the design of the residual block, which solves the gradient vanishing and gradient exploding problems in deep neural network training. The basic idea of the residual block is that it may be simpler to learn the residual between it and the input than to learn the direct output.

For each standard residual block, the formula is as follows:

F _out =l(f(F _in ,W _i )+F _in ) (1)

Where l represents the ReLU activation function, f is the weight operation in the residual block, W is the weight of the operation, and _Fin is the input of the residual block. _Fout represents the output of the residual block.

In step S3, an attention mechanism is added to the ResNet18 baseline model to increase the accuracy of gaze estimation. The structure of the facial feature extraction network is shown in the top branch of Figure 3, where Res1, Res2, Res3, and Res4 represent the four residual stages of resnet18, and each stage has a similar structure.

CBAM is a method that introduces an attention mechanism into convolutional neural networks. It is designed to enhance the adaptability of convolutional neural networks to spatial and channel information without significantly increasing computational complexity. The main idea of CBAM is to sequentially add two attention modules to the feature map: a channel attention module and a spatial attention module.

Channel attention aims to assign a weight to each channel, which is generally achieved through the features obtained by global average pooling and global maximum pooling. For a given feature map F∈R ^C×H×W , the calculation first calculates the global average pooling and global maximum pooling, then the channel attention can be expressed as these two values processed by a shared multi-layer perceptron and combined, the formula is as follows:

Where σ represents the Sigmoid activation function and M _CA represents channel attention. represents global average pooling, Represents global maximum pooling.

Spatial attention aims to assign a weight to each space. First, the global average pooling and global maximum pooling are still calculated, but this time along the channel dimension. Finally, the two feature maps are concatenated and passed through a 7×7 convolution layer, and then through a Sigmoid activation function. The formula is as follows:

M _SA (F)=σ(Conv ^7×7 ([F _gap ; F _gmp ])) (3)

Conv ^7×7 represents the 7×7 convolutional layer processing.

The face attention enhanced feature extraction module adds a CBAM attention module at the end of each stage to enhance the features, so that the module can emphasize more important feature channels and spatial positions, and also increase the ability to extract face features in low-resolution environments. For each residual stage, the formula can be expressed as follows:

Where ↓ represents downsampling. Then the features enhanced by the CBAM attention mechanism can be expressed as:

in represents the multiplication of corresponding elements, and F _FA represents the facial feature map after CBAM attention enhancement.

The structure of the first attention-modified residual stage is shown in Figure 4, and the remaining four stages adopt a similar structure.

In step S4, a local super-resolution method based on eye images is used to extract features, referred to as the LSRE module, which only performs super-resolution reconstruction on the eye images instead of the entire face image. The reason is that in line of sight estimation, the eye images contain the main line of sight related features and play a major role in line of sight estimation. Therefore, super-resolution only on the eye images can improve the efficiency of the model.

In step S4, FSRCNN is used as the network for super-resolution reconstruction. The FSRCNN algorithm is a deep convolutional neural network model designed for single image super-resolution. It is further improved on the basis of SRCNN in order to improve the speed and efficiency of super-resolution reconstruction.

The FSRCNN algorithm is mainly divided into three parts: feature extraction, contraction and expansion, and deconvolution. The first part is feature extraction, which uses a smaller convolution kernel of size 5×5 to extract features from low-resolution eye images. The purpose is to extract useful information from the original low-resolution image, which will then be used to reconstruct a high-resolution image. The formula for the feature extraction stage is as follows:

Where I _LR represents the low-resolution eye image, PRELU represents the PreLU activation function, and d represents the number of feature maps.

The second part is contraction and expansion. The contraction stage uses a 1×1 convolution kernel to reduce the number of feature maps. The purpose of this stage is to reduce the number of model parameters, thereby increasing the speed of operation. The mapping stage is to perform nonlinear mapping on the contracted feature space, which is achieved through multiple consecutive 3×3 convolution layers. The purpose of these layers is to capture high-level features of the image, thereby further enhancing the model's ability to capture details. The formula that can be expressed in the feature mapping stage is as follows:

The third part is deconvolution, which uses the deconvolution layer to expand the spatial size of the feature map to obtain a high-resolution output image. Different from the traditional bicubic interpolation method, the deconvolution at this stage is a learning method. This stage can convert the low-resolution feature map into the feature map of the high-resolution image. The formula is as follows:

F _SR = DeConv ^9×9 (F) (9)

Where DeConv represents the deconvolution operation.

In step S4, the eye feature extraction module DeepEyeNet is used to extract features, and the structure diagram of the module is shown in Figure 5. DeepEyeNet is a CNN network for eye deep feature extraction proposed by the present invention. It consists of ten convolution blocks, and the shallow feature extraction part of the network refers to the VGG16 network. The network has a total of 10 convolution blocks, each of which contains a Conv2d convolution layer, batch normalization, and ReLU. The maximum pooling is performed at the end of the 2nd, 5th, and 8th convolution blocks to reduce the size of the feature map. The output channels of the network convolution layer are 64*2, 128*3, 256*3, 512, and 1024, respectively. The last layer of the convolution layer passes through an adaptive average pooling layer to make the size of the output feature map become 1×1×1024. This is a relatively deep convolutional neural network. The spatial size of the feature map gradually decreases in the depth of the network. This deep network can better extract eye features after super-resolution to accurately estimate the line of sight. The left and right eyes go through a similar structure. Because the left and right eyes are similar, the two branches share weight information. The formula is as follows:

F′ _E =FC _E (FLAT(γ(F _SR ))) (10)

Where γ represents the feature extraction operation of DeepEyeNet, _FE ′ represents the output eye feature, and FC _E represents the eye fully connected operation.

After obtaining the features of the face and eyes, the network passes through a fully connected layer to connect the features of the face, left eye, and right eye, and outputs the final two-dimensional features as the line of sight estimation result.

ξ _pred =FC _T ([FC _FA (F _FA ); F′ _E ]) (11)

Where FC _FA represents the fully connected operation of the face.

In step S4, MSELoss is used as the loss function for line of sight estimation. MSELoss is also called mean square error, which represents the expected value of the square of the error between the model's estimated prediction value and the true value. The smaller the MSELoss, the larger the error. Therefore, the loss function for line of sight estimation in this module is:

Among them, the true value of the line of sight is ξ _gt , and the predicted value of the line of sight estimation is ξ _pred .

In order to evaluate the performance of the proposed network model, the present invention trains and evaluates the model on the well-known public dataset MPIIFaceGaze for gaze estimation tasks. The MPIIFaceGaze dataset was proposed by Zhang et al. This dataset contains images from 15 different subjects. These images are collected under different head deflection directions, different head postures, and different lighting conditions. The evaluation subset provided is 3,000 samples randomly selected from each subject's image, for a total of 45,000 samples. Like other works, the present invention follows the evaluation method provided by the dataset, that is, one-person-leave-one-out cross-validation is used for evaluation.

The present invention follows the data processing method proposed by Zhang et al., the author of the dataset, to perform data preprocessing. At the same time, in order to simulate the LR environment, the present invention downsamples the dataset and simulates different resolution environments, where the high-resolution (HR) image size is 128×128. The present invention uses different scale factors (2×, 4×, 8×) to perform bicubic downsampling on the HR image to obtain three groups of LR images, whose resolutions are 64×64, 32×32, and 16×16, respectively. These four groups of datasets are used for experiments to improve the robustness of the experiment and to comprehensively evaluate the experimental results at different resolutions. The evaluation is also performed using leave-one-out cross-validation.

In order to verify the effectiveness of the LOSRG-Net network proposed in this paper, an experimental comparison is conducted with other advanced methods on the MPIIFaceGaze dataset. In order to make a fair comparison, the experimental settings of each method adopt the settings in the corresponding paper, including the model framework and hyperparameters, so as to reproduce their network performance.

The performance comparison of the proposed LOSRG-Net and these advanced methods on the dataset MPIIFaceGaze is shown in Table 1. The top row indicates the resolution of the dataset, which are 128×128, 64×64, 32×32, and 16×16, respectively, indicating the results from high to low resolutions, and each column indicates the angular error of different models at different resolutions.

Table 1 Comparative experimental results of the present invention and other advanced methods on the MPIIFaceGaze dataset

As shown in Table 1, the angle error of the LOSRG-Net method is smaller than that of all the advanced methods in each resolution scenario, which means that the method of the present invention achieves the best performance. At the same time, the performance of almost all models will decrease as the resolution decreases, and some models will significantly reduce the performance when the resolution decreases. However, the performance of the method of the present invention deteriorates the slowest as the resolution of the input image decreases. The method of the present invention has outstanding effects in extremely low-resolution scenarios, indicating that it can effectively achieve line of sight estimation in low-resolution scenarios.

This embodiment will introduce an applicable scenario of the present invention:

Line of sight estimation has a wide range of application scenarios, one of which is the monitoring of online learning concentration. At present, online teaching has become popular and appears in many teaching occasions. The disadvantage of online teaching compared to offline teaching is that there is no supervision, and students are prone to fatigue or distraction, resulting in inattention in class, which will affect the quality of education and teaching effect. The method proposed in this invention can effectively solve this problem. The specific steps are as follows:

S1, extracting frame image data of online learner video stream data;

S2, extracting facial features from the frame image data, and locating the face area and eye area;

S3, passing the facial image and binocular image through the sight line estimation model of the present invention to obtain the sight line estimation direction;

S4, calculating the concentration according to the sight line estimation direction result obtained by the present invention, setting the threshold of the attention detection module, if it is greater than or equal to the threshold, it is judged as distracted, and if it is less than the threshold, it is judged as focused;

S5. Collect the online learning video stream data of the online learner in real time, and then execute step S1 to step S2; then execute step S3 and apply the sight estimation model to obtain the eye sight direction, and then apply the attention detection module of step S4 to determine the attention of the online learner.

S6. The system sends a warning to the learner, reminding him to concentrate.

The present invention proposes a line of sight estimation method based on local super-resolution fused attention mechanism, and designs an eye feature extraction module based on local super-resolution. The network first reconstructs the eye image with super-resolution to efficiently and quickly restore the low-resolution line of sight related features, and then passes through a novel deep convolutional neural network. The spatial size of the feature map gradually decreases in the depth of the network. This network can better extract the eye features after super-resolution to accurately estimate the line of sight. The present invention also proposes a face attention enhanced feature extraction network, which fuses the attention mechanism to enhance the low-resolution global features from both spatial and channel directions to increase the ability to extract facial features in a low-resolution environment, thereby improving the effect of line of sight estimation. After experimental verification, this method can effectively improve the accuracy of line of sight estimation in low-resolution scenes.

The specific implementation scheme described above further describes in detail the purpose, technical scheme and beneficial effects of the present invention. It should be understood that the above is only a specific implementation scheme of the present invention and is not intended to limit the scope of the present invention. Any equivalent changes and modifications made by any technician in the field without departing from the concept and principle of the present invention should fall within the scope of protection of the present invention.

Claims (5)

1. A sight line estimation method based on a local super-resolution fusion attention mechanism is characterized by comprising the following steps:

s1, acquiring a frame image by using a camera;

S2, detecting and positioning a face area and a binocular area by adopting a face detection model, cutting a face image, and intercepting an eye image;

S3, strengthening and extracting the facial image features through a facial attention strengthening feature extraction module;

s4, extracting features of the binocular image through an eye feature extraction module based on local super resolution;

S5, obtaining a sight estimation result through fusion of the extracted face image and the binocular image features by the full-connection layer;

In step S3, the facial attention enhancing feature extraction module uses ResNet model as the reference model of the facial feature extraction module, and for each standard residual block, the formula is as follows:

F_out＝l(f(F_in,W_i)+F_in) (1)

Where l represents the ReLU activation function, F is the weight operation in the residual block, W is the weight of the operation, F _in is the input of the residual block, and F _out represents the output of the residual block;

In step S4, the local super-resolution-based eye feature extraction module uses FSRCNN as a super-resolution reconstruction network;

the FSRCNN algorithm mainly comprises three parts of feature extraction, contraction, expansion and deconvolution; the first part is feature extraction, which uses a smaller convolution kernel, with a size of 5×5, to extract features from low resolution eye images, and the formula of the feature extraction stage is as follows:

Wherein I _LR represents a low-resolution eye image, PRELU represents a PreLU activation function, and d represents the number of feature maps;

The second part is contraction and expansion, wherein a1×1 convolution kernel is used in the contraction stage to reduce the number of feature maps, and the mapping stage is to perform nonlinear mapping on the contracted feature space and is implemented by a plurality of continuous 3×3 convolution layers, wherein the feature mapping stage is expressed by the following formula:

The third part is deconvolution, which uses deconvolution layers to expand the spatial size of the feature map, converting the low resolution feature map into a feature map of the high resolution image, as shown below:

F_SR＝DeConv^9×9(F) (9)

wherein DeConv denotes a deconvolution operation.

2. The line-of-sight estimation method based on local super-resolution fusion attention mechanism according to claim 1, wherein in step S2, the face detection model adopts a face detection model based on a convolutional neural network.

3. The line-of-sight estimation method based on local super-resolution fusion attention mechanism according to claim 2, wherein in step S3, the face attention enhancement feature extraction module adds CBAM attention mechanism on the basis of ResNet a 18 reference model; CBAM attention mechanism two attention modules are added to the feature map: a channel attention module and a spatial attention module;

The channel attention module aims at distributing a weight to each channel, and is realized through the characteristics obtained by global average pooling and global maximum pooling; for a given feature map F ε R ^C×H×W, first compute global average pooling and global maximum pooling, and channel attention is expressed as the two values are processed by a shared multi-layer perceptron and combined as follows:

where σ represents the Sigmoid activation function, M _CA represents the channel attention, Representing a global average pooling of the data,Representing global maximum pooling;

The spatial attention module aims to assign a weight to each space, first still to compute global average pooling and global maximum pooling, but this time along the dimensions of the space, finally concatenate the two feature maps and pass through a 7 x 7 convolution layer, then activate the function by a Sigmoid, formulated as follows:

M_SA(F)＝σ(Conv^7×7([F_gap;F_gmp])) (3)

Wherein Conv ^7×7 denotes a 7 x 7 convolutional layer process;

The human face attention enhancement feature extraction module adds CBAM attention modules at the end of each stage to enhance the features, and for each residual stage, the expression formula is as follows:

wherein ∈represents downsampling; the features after CBAM attention mechanism enhancement are expressed as follows:

wherein, Representing the multiplication of the corresponding elements, and F _FA represents the attention-enriched facial feature map of CBAM.

4. The sight line estimation method based on the local super-resolution fusion attention mechanism according to claim 3, wherein in step S4, the local super-resolution based eye feature extraction module extracts features by using eye feature extraction DEEPEYENET; DEEPEYENET is an eye depth feature extraction CNN network, which is composed of ten convolution blocks, and is a convolution neural network with relative depth, the spatial dimension of feature mapping gradually decreases in the depth of the network, and the left eye and the right eye are in similar structures, and the formula is as follows:

F_E′＝FC_E(FLAT(γ(F_SR))) (10)

Wherein γ represents the feature extraction operation of DEEPEYENET, F _E' represents the output eye feature, and FC _E represents the eye full connection operation;

after the characteristics of the face and the eyes are obtained, the network is connected with the characteristics of the face, the left eye and the right eye through a full connection layer, and the final two-dimensional characteristics are output as a sight line estimation result:

ξ_pred＝FC_T([FC_FA(F_FA);F_E′]) (11)

Where FC _FA represents a face full join operation.

5. The line-of-sight estimation method based on local super-resolution fusion attention mechanism according to claim 4, wherein MSELoss is adopted as a loss function of line-of-sight estimation, and the loss function of line-of-sight estimation is:

Wherein the true value of the line of sight is ζ _gt and the predicted value of the line of sight estimate is ζ _pred.