CN118865336A - A lightweight YOLOv8-pose based method for fatigue driving detection - Google Patents

Abstract

The present invention proposes a lightweight YOLOv8-pose fatigue driving detection method. The method constructs a lightweight YOLOv8-pose model with a collected multi-pose and multi-view face data set, introduces Ghost convolution in the model backbone network to reduce the model parameter amount and unnecessary convolution calculation, accelerates network prediction calculation by introducing different size features extracted by the Slim-neck fusion model backbone network, adds an occlusion-aware attention module SEAM to the Neck part of the model to emphasize the face area in the image and weaken the background, so as to improve the key point positioning effect, and adopts a GNSC-Head structure in the detection head part of the model, which uses shared convolution to optimize the convolutional BN layer into a more stable GN layer, so as to save the parameter space and computing resources of the model; the method constructs a fatigue decision model and evaluates the output result of the model to judge whether the driver is in a fatigue state; the method of the present invention is conducive to reducing the computational complexity and improving the fatigue detection accuracy.

Description

A lightweight YOLOv8-pose based method for fatigue driving detection

Technical Field

The present invention relates to the technical field of fatigue driving detection, and in particular to a lightweight YOLOv8-pose fatigue driving detection method.

Background Art

In recent years, with the development of social economy, people's living standards have gradually improved, and the per capita car ownership has increased. However, with the rapid growth of the number of cars and road mileage, traffic accidents have also increased to a certain extent, of which about 18% are caused by driver fatigue. According to investigations and studies, if certain reminders and warnings can be given when the driver is fatigued, 90% of similar traffic accidents can be avoided. Therefore, studying an efficient fatigue driving detection method is of great significance to traffic safety.

At present, the research on fatigue driving detection is mainly divided into contact detection and non-contact detection. Contact detection is to obtain the driver's physiological information and conduct research by wearing sensors. Generally, the driver's fatigue level is evaluated based on EEG signals and ECG signals. Although the recognition rate is high, it requires wearing complex hardware equipment, so it is difficult to popularize and apply. Non-contact mainly uses two methods. The first is to detect fatigue driving based on vehicle trajectory. Although it does not require direct contact with the driver, it is easily affected by road conditions and weather, and the accuracy is low. The second is to use image acquisition equipment to collect information on the driver's face and determine the driver's fatigue state through a computer. Compared with physiological signal acquisition equipment and vehicle trajectory acquisition equipment, surveillance cameras are low-cost, highly convenient, and do not affect the driver's normal driving. Therefore, this type of research has more application potential.

However, existing deep learning-based fatigue detection algorithms have some shortcomings. Some methods only use a single feature for fatigue detection, without considering the driver's various fatigue states, and have poor robustness. Although some methods have high detection accuracy, the model parameters and calculations used are large, making it difficult to deploy in vehicle-mounted edge devices. Although some methods use lightweight models, the number of detection models used is too large, and the separated models need to be trained separately, which cannot be optimized end-to-end. There is also a certain delay in the transmission between models, resulting in poor real-time performance of fatigue detection.

Summary of the invention

The present invention proposes a lightweight YOLOv8-pose fatigue driving detection method, which is beneficial to reducing computational complexity and improving fatigue detection accuracy.

The present invention adopts the following technical solutions.

A lightweight YOLOv8-pose fatigue driving detection method is disclosed. The method uses a collected multi-pose and multi-view face data set to construct a lightweight YOLOv8-pose model, introduces Ghost convolution in the model backbone network to reduce the model parameters and unnecessary convolution calculations, accelerates network prediction calculations by introducing different size features extracted by the Slim-neck fusion model backbone network, adds an occlusion-aware attention module SEAM to the Neck part of the model to emphasize the face area in the image and weaken the background to improve the key point positioning effect, and adopts a GNSC-Head structure in the detection head part of the model, which uses shared convolution to optimize the convolutional BN layer into a more stable GN layer to save the model's parameter space and computing resources; the method constructs a fatigue decision model and evaluates the output result of the model to determine whether the driver is in a fatigue state.

The method comprises the following steps:

Step S1: Collect multi-pose and multi-view face datasets, convert the annotation files, and generate corresponding YOLO annotation files;

Step S2: Build and train a lightweight YOLOv8-pose model, use the trained model to perform face detection on the input image, and locate facial key points after recognizing the face. Finally, perform classification and regression in the detection head to obtain the key point coordinates of the driver's eyes and mouth.

Step S3: construct a fatigue decision model, give fatigue judgment based on the features of eyes and mouth, use PERCLOS evaluation criteria to judge whether the eye information is open or closed, and judge whether the driver is in a fatigue state by detecting the number of times the driver's eyes are closed within a period of time; the evaluation criteria for mouth features adopt the same principle method, that is, judge whether the mouth is open or closed based on the mouth coordinate information, and judge whether the driver is in a fatigue state by detecting the number of times the driver yawns within a period of time;

Step S4: Provide a visual warning for the judgment results of the driver's fatigue behavior obtained by the identification and analysis in steps S2 and S3.

In step S1, the AFLW dataset is introduced as a large-scale face dataset containing multiple poses and multiple perspectives. If the number of closed-eye images in the dataset is too low to meet the demand, the CEW closed-eye dataset is introduced as a supplement. When the format of the annotation file in the above dataset is not the annotation format of the YOLO model, the annotation file is converted by processing the data using Python to generate the corresponding YOLO annotation file. In step S1, the above dataset is randomly sampled in a ratio of 6:2:2 and divided into a training set, a validation set, and a test set.

In step S2, the lightweight YOLOv8-pose introduces a lightweight convolution GhostConv in the backbone network to use cheap linear transformation to generate a large number of Ghost feature maps that can mine the required information from the original features at a low cost, so as to reduce the number of parameters and calculations of the model;

In step S2, a C3 module with fast detection speed is introduced. By performing lightweight operations on the C3 module, two identical GhostConvs are combined and connected to form a Ghost-Bottleneck structure. The reorganized C3 module is used as the C3Ghost module to reduce the complexity of the model and the amount of calculation of the facial key point model.

In step S2, the lightweight YOLOv8-pose uses the Slim-neck network structure as the enhanced feature fusion network, introduces the lightweight convolution GSConv, and uses dense convolution calculations to maximize the implicit connections between each channel to accelerate the prediction calculation of the model. The GS-bottleneck structure is formed through residual connections to further enhance the network's ability to process features; finally, the VOV-GSCSP module is formed through a one-time aggregation method, so that gradients at different positions can be cross-mixed to enhance the gradient performance and learning ability of the network.

In step S2, the lightweight YOLOv8-pose adds occlusion-aware attention SEAM to emphasize the face area in the image and weaken the background while achieving multi-scale face detection, thereby improving the facial landmark location effect.

In step S2, the lightweight YOLOv8-pose retains the decoupled structure with the GNSC-Head detection head, and uses group normalized shared convolution to make lightweight improvements to the network.

In step S2, a lightweight YOLOv8-pose key point detection model is built using the Pytorch deep learning framework. The number of training rounds is set to 300, the input image size is set to 640×640, the batch size is 64, the training weights are saved every 10 rounds, and the mAP is calculated using the validation set to evaluate the model performance. Finally, the model with the largest mAP is selected as the final model.

In step S3, the eye aspect ratio EAR is calculated based on the eye key point information output by the lightweight YOLOv8-pose. The EAR threshold is set with reference to the P80 standard in the PERCLOS standard. P80 means that the eye is considered closed when the area of the eyelid covering the pupil exceeds 80%. The formula is as follows:

Where x _i represents the horizontal coordinate of the eye key point, y _i represents the vertical coordinate, _fe is defined to judge the driver's eye fatigue state, _te is the number of closed-eye frames within the detection time, _{and Te} is the total number of frames within the detection time;

In step S3, the mouth aspect ratio MAR is calculated based on the mouth key point information output by the lightweight YOLOv8-pose. The formula is as follows:

Where _xi represents the horizontal coordinate of the mouth key point, _yi represents the vertical coordinate, _fm is defined to judge the driver's mouth fatigue state, _tm represents the number of mouth opening frames within the detection time, and _Tm represents the total number of frames within the detection time.

In step S3, the output results of the model are evaluated by the algorithm, and various indicators are combined to determine whether the driver is in a fatigue state, and a visual warning is issued. The triggering criteria of the visual warning are:

Under normal conditions, the closing time of one eye is 0.1-0.15 seconds, while under fatigue conditions, the closing time of one eye will be greater than 0.5 seconds. That is, if the closing frequency per unit time is greater than 0.5, it is judged that the driver is in a fatigue state.

Usually, the duration of a human yawn is 3-5 seconds. In the algorithm of step S3, the unit detection time is selected as 30 seconds, and the number of yawns within 30 seconds cannot exceed twice, that is, when the yawn frequency per unit time is greater than 0.4, it is judged that the driver is in a fatigue state.

Compared with the prior art, the present invention has the following gain effects: The present invention provides a lightweight YOLOv8-pose fatigue driving detection method, which uses a lightweight YOLOv8-pose model to realize face detection and facial key point positioning, and judges the driver's mental state through a fatigue decision module. The lightweight YOLOv8-pose model uses Ghost convolution and GS convolution to greatly reduce the number of network parameters and improve the detection rate of the model. At the same time, it adopts an occlusion-aware attention mechanism and a GNSC-Head structure to maintain high detection accuracy while being lightweight. The fatigue decision module uses a multi-feature fatigue judgment method to more effectively identify the driver's state, providing strong support for the deployment of vehicle edge devices.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

FIG1 is a flowchart of a method implementation of an embodiment of the present invention;

Figure 2 is a structural diagram of a key point detection model in an embodiment of the present invention;

FIG3 is a structural diagram of a SEAM module in an embodiment of the present invention;

FIG4 is a structural diagram of a GNSC-Head module in an embodiment of the present invention;

FIG. 5 is a distribution diagram of key points of a face in an embodiment of the present invention;

FIG6 is a visualized schematic diagram of a fatigue detection system in an embodiment of the present invention.

DETAILED DESCRIPTION

The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in the present invention have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and/or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and/or combinations thereof.

As shown in the figure, a lightweight YOLOv8-pose fatigue driving detection method is provided. The method constructs a lightweight YOLOv8-pose model with a collected multi-pose and multi-view face dataset, introduces Ghost convolution in the model backbone network to reduce the model parameters and unnecessary convolution calculations, accelerates network prediction calculations by introducing different size features extracted by the Slim-neck fusion model backbone network, adds an occlusion-aware attention module SEAM to the Neck part of the model to emphasize the face area in the image and weaken the background to improve the key point positioning effect, and adopts a GNSC-Head structure in the detection head part of the model, which uses shared convolution to optimize the convolutional BN layer into a more stable GN layer to save the model's parameter space and computing resources; the method constructs a fatigue decision model and evaluates the output results of the model to determine whether the driver is in a fatigue state.

The method comprises the following steps:

Step S1: Collect multi-pose and multi-view face datasets, convert the annotation files, and generate corresponding YOLO annotation files;

Step S2: Build and train a lightweight YOLOv8-pose model, use the trained model to perform face detection on the input image, and locate facial key points after recognizing the face. Finally, perform classification and regression in the detection head to obtain the key point coordinates of the driver's eyes and mouth.

Step S3: construct a fatigue decision model, give fatigue judgment based on the features of eyes and mouth, use PERCLOS evaluation criteria to judge whether the eye information is open or closed, and judge whether the driver is in a fatigue state by detecting the number of times the driver's eyes are closed within a period of time; the evaluation criteria for mouth features adopt the same principle method, that is, judge whether the mouth is open or closed based on the mouth coordinate information, and judge whether the driver is in a fatigue state by detecting the number of times the driver yawns within a period of time;

Step S4: Provide a visual warning for the judgment results of the driver's fatigue behavior obtained by the identification and analysis in steps S2 and S3.

In step S1, the AFLW dataset is introduced as a large-scale face dataset containing multiple poses and multiple perspectives. If the number of closed-eye images in the dataset is too low to meet the demand, the CEW closed-eye dataset is introduced as a supplement. When the format of the annotation file in the above dataset is not the annotation format of the YOLO model, the annotation file is converted by processing the data using Python to generate the corresponding YOLO annotation file. In step S1, the above dataset is randomly sampled in a ratio of 6:2:2 and divided into a training set, a validation set, and a test set.

In step S2, the lightweight YOLOv8-pose introduces a lightweight convolution GhostConv in the backbone network to use cheap linear transformation to generate a large number of Ghost feature maps that can mine the required information from the original features at a low cost, so as to reduce the number of parameters and calculations of the model;

In step S2, a C3 module with fast detection speed is introduced. By performing lightweight operations on the C3 module, two identical GhostConvs are combined and connected to form a Ghost-Bottleneck structure. The reorganized C3 module is used as the C3Ghost module to reduce the complexity of the model and the amount of calculation of the facial key point model.

In step S2, the lightweight YOLOv8-pose uses the Slim-neck network structure as the enhanced feature fusion network, introduces the lightweight convolution GSConv, and uses dense convolution calculations to maximize the implicit connections between each channel to accelerate the prediction calculation of the model. The GS-bottleneck structure is formed through residual connections to further enhance the network's ability to process features; finally, the VOV-GSCSP module is formed through a one-time aggregation method, so that gradients at different positions can be cross-mixed to enhance the gradient performance and learning ability of the network.

In step S2, the lightweight YOLOv8-pose adds occlusion-aware attention SEAM to emphasize the face area in the image and weaken the background while achieving multi-scale face detection, thereby improving the facial landmark location effect.

In step S2, the lightweight YOLOv8-pose retains the decoupled structure with the GNSC-Head detection head, and uses group normalized shared convolution to make lightweight improvements to the network.

In step S2, a lightweight YOLOv8-pose key point detection model is built using the Pytorch deep learning framework. The number of training rounds is set to 300, the input image size is set to 640×640, the batch size is 64, the training weights are saved every 10 rounds, and the mAP is calculated using the validation set to evaluate the model performance. Finally, the model with the largest mAP is selected as the final model.

In step S3, the eye aspect ratio EAR is calculated based on the eye key point information output by the lightweight YOLOv8-pose. The EAR threshold is set with reference to the P80 standard in the PERCLOS standard. P80 means that the eye is considered closed when the area of the eyelid covering the pupil exceeds 80%. The formula is as follows:

Where x _i represents the horizontal coordinate of the eye key point, y _i represents the vertical coordinate, _fe is defined to judge the driver's eye fatigue state, _te is the number of closed-eye frames within the detection time, _{and Te} is the total number of frames within the detection time;

In step S3, the mouth aspect ratio MAR is calculated based on the mouth key point information output by the lightweight YOLOv8-pose. The formula is as follows:

Where _xi represents the horizontal coordinate of the mouth key point, _yi represents the vertical coordinate, _fm is defined to judge the driver's mouth fatigue state, _tm represents the number of mouth opening frames within the detection time, and _Tm represents the total number of frames within the detection time.

In step S3, the output results of the model are evaluated by the algorithm, and various indicators are combined to determine whether the driver is in a fatigue state, and a visual warning is issued. The triggering criteria of the visual warning are:

Under normal conditions, the closing time of one eye is 0.1-0.15 seconds, while under fatigue conditions, the closing time of one eye will be greater than 0.5 seconds. That is, if the closing frequency per unit time is greater than 0.5, it is judged that the driver is in a fatigue state.

Usually, the duration of a human yawn is 3-5 seconds. In the algorithm of step S3, the unit detection time is selected as 30 seconds, and the number of yawns within 30 seconds cannot exceed twice, that is, when the yawn frequency per unit time is greater than 0.4, it is judged that the driver is in a fatigue state.

Example:

In this case, AFLW is a large-scale face database with multiple poses and perspectives, which contains photos affected by different poses, expressions, lighting, race and other factors. The database has about 25,000 manually annotated face pictures, which are suitable for face recognition, face key point detection and other fields. Since there are too few closed-eye pictures in the data set, the CEW closed-eye data set is introduced as a supplement, which contains 2423 photos of testers with open and closed eyes. The differentiation of photos is reflected in the individual differences of the testers and the changes in various environments, such as lighting, blur, occlusion and other factors, which can effectively improve the robustness of model training.

In this example, the key point detection model constructed for fatigue detection is a lightweight YOLOv8-pose model, and its model structure is shown in Figure 2.

Specifically, the backbone network is used for image feature extraction, which consists of Ghost convolution and C3Ghost modules. In deep convolutional neural networks, the output feature maps of the intermediate layers usually contain rich or even redundant feature maps, and the feature information of some feature maps is relatively similar, which will take up a lot of memory and FLOPs. Ghost convolution generates many Ghost feature maps that can mine the required information from the original features at a very low cost through cheap linear transformation, which can effectively reduce the number of parameters and calculations of the model. In the original YOLOv8n-pose backbone network, the C2f module sacrifices a certain speed in order to obtain more rich gradient flow information, but in the fatigue driving detection system, since the surrounding scenes are relatively simple and there is no need to identify smaller targets, the C2f module is optimized to the C3 module, and a lightweight operation is performed on the C3 module to introduce the Ghost-Bottleneck structure in the Ghost-net network, in which the first Ghost convolution is used to compress the number of channels, reduce the number of parameters, and reduce the impact of high-frequency noise. The second Ghost convolution is used to restore the number of channels. Then, the original features are added through the residual connection operation to supplement the information loss caused by channel compression. Such a bottleneck structure is conducive to reducing the amount of parameters and information loss. At the same time, in order not to affect the feature extraction ability of the backbone network, the remaining convolution operations in the C3 module remain unchanged. The improved C3 module is named C3Ghost module. As a lightweight network, the C3Ghost module can effectively reduce the number of network parameters and computational costs while maintaining the size of the original output feature map and the channel size, and further reduce the complexity of the model and the amount of computation of the face key point model.

The function of the feature fusion network is to enhance the extraction of the output features of the backbone network, and to connect the features extracted at different stages horizontally, so as to achieve the fusion of high-level semantic features and low-level detail features, thereby generating a richer feature expression. In order to further realize the lightweight of the model without affecting the model's ability to extract features, the present invention introduces GSConv in Slim-neck and uses the VoV-GSCSP module to improve the Neck of YOLOv8-pose, further optimizing the parameter quantity and computational complexity of the model. GSConv retains the implicit connection between each channel to the maximum extent through dense convolution calculation, avoids the loss of semantic information caused by each spatial compression and channel expansion of the feature map, and accelerates the prediction calculation of the model. And the GS-bottleneck structure is formed by residual connection to further enhance the network's ability to process features. Finally, the VOV-GSCSP module is formed by a one-time aggregation method, so that the gradients at different positions can be cross-mixed, enhancing the gradient performance and learning ability of the network, and ensuring the accuracy of the model while being lightweight.

At the same time, the Separated and Enhancement Attention Module (SEAM) is added to the feature fusion network. This module can effectively avoid the problem of low face recognition rate and inaccurate positioning caused by the driver wearing sunglasses or masks while driving. The overall architecture of SEAM is shown in Figure 3. The left side is the architecture of SEAM, and the right side is the structure of DcovN (channel and spatial hybrid module). The first part of DcovN is a depth-wise convolution with residual connection. Although the use of depth-wise convolution can learn the importance of different channels and reduce the number of parameters, it ignores the information relationship between channels. In order to make up for this loss, the point-by-point convolution is introduced in the second part to enhance the representation and generalization capabilities of the DcovN module. After the input feature map passes through the DcovN module, it passes through the average pooling layer to reduce the spatial size of the feature map and retain important feature information. Then two fully connected networks are used to fuse the information of each channel, so that the network can strengthen the connection between all channels. The output learned by the fully connected layer is processed by an exponential function to expand the value range from [0,1] to [1,e]. Since the normalization of the exponential can provide a monotonic mapping relationship, the output result can tolerate position errors better. Finally, the output of the SEAM module is multiplied by the original feature, so that the model can effectively handle the occluded part of the face.

The YOLOv8-pose detection head adopts a decoupled structure, which extracts the target position, category information and key point coordinates respectively, learns them separately through different network branches, and finally fuses them. Compared with the traditional coupling head, the decoupling head can better handle semantic information of different scales and fineness, and improve the generalization ability and robustness of the model. But at the same time, the number of parameters is also greatly increased. The number of parameters of the detection head alone accounts for 1/3 of the entire model. Therefore, in order to improve the detection speed of the model while maintaining the stability of accuracy, the present invention adopts the GNSC-Head detection head, retains the decoupled structure of the original detection head, and uses group normalized shared convolution to improve the network. The improved detection head network structure is shown in Figure 4.

GNSC-Head introduces shared convolution and uses it in multiple locations, saving parameter space and computing resources, so that it can be deployed and applied in vehicle systems with limited space. At the same time, in order to make up for the decline in the model's feature extraction ability caused by the use of shared convolution, the BN layer of the convolution layer is replaced by the GN layer. Compared with the BN layer, GN divides the feature map into several groups and normalizes each group. It does not depend on the batch size, so it can maintain good performance even in the case of small batches of high-precision images. At the same time, in the detection task, the input of the detection head comes from the ROI area, which is sampled from the same image. They do not meet the independent and identically distributed assumption, and non-independent and identically distributed will weaken the mean and variance distribution of the BN layer. Therefore, in the detection head, using GN will have a better effect than using BN.

In the classification task, no matter the detection layer of large, medium or small targets, they all detect the same target, so shared convolution can better adapt to the different stages of the classification task, thereby improving the generalization ability of the model on targets of different scales. In the bounding box regression task, since the features extracted by shared convolution are the same for targets of all scales, it is impossible to effectively distinguish targets of different scales. For this reason, a Scale layer is added at the end of the network. By introducing a learnable scaling factor, the model is helped to better extract features on targets of different scales, thereby effectively adjusting the scale of the input. In the key point regression task, the number of channels of the shared convolution needs to be divisible by the number of groups to ensure that each group has the same number of channels, but the total number of face key points in the present invention cannot be divided by the number of groups, and the key point regression task needs to accurately locate the key points of the target image, which is a relatively fine-grained task. Therefore, the network structure of the original detection head is selected in this task to ensure the overall accuracy of the model.

Step S3 in this example is specifically as follows: construct a fatigue decision model, make fatigue judgments based on the features of the eyes and mouth, use the PERCLOS evaluation criteria to determine whether the eye information is open or closed, and determine whether the driver is in a fatigued state by detecting the number of times the driver's eyes are closed within a period of time. The evaluation criteria for mouth features are similar, and the mouth coordinate information is determined to be open or closed, and the number of times the driver yawns within a period of time is detected to determine whether the driver is in a fatigued state.

Specifically, the image is input through the lightweight YOLOv8n-pose network model, the facial key points are extracted after the facial area is detected from the image, and the positions of the key points are drawn in the image. The traditional facial key points are composed of 68 points, as shown in Figure 5, but in the actual detection process, only the driver's eyes and mouth parts are obtained to determine whether the driver is tired. Therefore, the invention only selects 18 of the facial key points for fatigue status assessment, namely 12 key points (37-48) for the left and right eyes, and 6 key points (49, 51, 53, 55, 57, 59) for the mouth.

The x and y coordinates of the key points of the eye are used to obtain the Euclidean distance of each point, and the aspect ratio (EAR) of the eye is calculated. The threshold of EAR refers to PERCLOS, which is an internationally recognized standard for judging fatigue. It refers to the proportion of time when the eyes are closed within a certain period of time. The PERCLOS judgment standard includes P70, P80 and EM, which respectively indicate that the area of the pupil covered by the eyelid is more than 70%, 80% and 50% and is considered as eye closure. Among them, P80 is considered to be the most sensitive standard for fatigue. The eye status evaluation index in this article refers to the P80 standard in the PERCLOS standard, that is, the EAR threshold is set at 0.2. When the EAR is less than 0.2, the eyes are judged to be closed. The formula is as follows:

The driver's eye fatigue state is determined by defining _fe , where _te is the number of closed eye frames within the detection time, and _Te is the total number of frames within the detection time. Under normal conditions, the single eye closure time is usually 0.1-0.15 seconds, while under fatigue conditions, the single eye closure time is usually greater than 0.5 seconds. Therefore, the parameter threshold of _fe is set to 0.5, that is, when the eye closure frequency per unit time is greater than 0.5, the driver is judged to be in a fatigue state.

Similarly, the mouth state evaluation index is similar to the eye state evaluation index. The x and y coordinates of the key points of the mouth are used to obtain the Euclidean distance of each point, calculate the length-width ratio (MAR) of the mouth, and define f _m to judge the driver's mouth fatigue state. The calculation formula is as follows:

Where _tm represents the number of mouth-opening frames within the detection time, and _Tm represents the total number of frames within the detection time. Usually, the duration of a human yawn is 3-5 seconds. In this algorithm, the unit detection time is selected as 30 seconds, and the number of yawns within 30 seconds cannot exceed twice. Therefore, this paper sets the parameter threshold of _fm to 0.4, that is, when the yawning frequency per unit time is greater than 0.4, it is judged that the driver is in a fatigue state.

S4: Provide visual warning for the judgment results of driver fatigue behavior obtained through identification and analysis in S2 and S3.

Specifically, in order to test the generalization ability and effectiveness of the lightweight YOLOv8-pose model in the driving environment, this paper uses the recognition rate of the YawnDD (YAWNING DETECTION DATASET) driving detection video dataset as the judgment indicator. The video data is collected from the main driver's seat of the car, including forward video collection and oblique side video collection. The video content captured includes normal driving, talking, and fatigue yawning. According to the above method, EAR and MAR are calculated to detect the number of times the driver closes his eyes and yawning behavior. As shown in Figure 6, the lightweight YOLOv8-pose model can accurately detect and track the driver's blinking and yawning behavior during driving and perform effective recognition statistics and driving state discrimination. Even when the female driver wears glasses or the male driver drives in a dark environment, the model can still identify the fatigue state of the drivers.

In summary, the lightweight YOLOv8-pose fatigue driving detection method described in the present invention can effectively improve the model algorithm's feature extraction capability and detection accuracy for key points, improve the lightweight performance of the model algorithm, and greatly improve the real-time performance of the system.

Many specific details are described in the above description to facilitate a full understanding of the present invention. However, the above description is only a preferred embodiment of the present invention. The present invention can be implemented in many other ways different from those described herein, so the present invention is not limited to the specific implementation disclosed above. At the same time, any person familiar with the art can make many possible changes and modifications to the technical solution of the present invention using the methods and technical contents disclosed above without departing from the scope of the technical solution of the present invention, or modify it into an equivalent embodiment of equivalent changes. Any simple modification, equivalent change and modification made to the above embodiments according to the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of protection of the technical solution of the present invention.

Claims (10)

1. A fatigue driving detection method for light YOLOv-pose is characterized in that: according to the method, a lightweight YOLOv-pose model is built by an acquired multi-pose and multi-view face data set, a Ghost convolution is introduced into a model backbone network to reduce the number of model parameters and unnecessary convolution calculation, network prediction calculation is accelerated by introducing different size features extracted by a Slim-neg fusion model backbone network, a shielding perception attention module SEAM is added in a model Neck part to emphasize a face region in an image and weaken a background so as to improve a key point positioning effect, and a GNSC-Head structure is adopted in a detection Head part of the model, and a shared convolution is used to optimize a convolved BN layer into a more stable GN layer so as to save parameter space and calculation resources of the model; the method comprises the steps of constructing a fatigue decision model and evaluating an output result of the model to judge whether a driver is in a fatigue state.

2. The method for detecting fatigue driving of light YOLOv-pose according to claim 1, wherein the method comprises the steps of: the method comprises the following steps;

Step S1: collecting a multi-gesture and multi-view face data set, and converting the annotation file to generate a corresponding YOLO annotation file;

Step S2: constructing and training a lightweight YOLOv-pose model, adopting the trained model to detect the human face of an input image, positioning key points of the face after the human face is identified, and finally classifying and regressing the detected head part to obtain the coordinates of the key points of eyes and a mouth of a driver;

Step S3: constructing a fatigue decision model, giving out fatigue judgment aiming at the characteristics of eyes and a mouth, adopting a PERCLOS evaluation criterion to carry out eye opening and closing judgment on eye information, and judging whether a driver is in a fatigue state or not by detecting the closing times of eyes of the driver within a period of time; the evaluation criterion of the mouth characteristics adopts the same principle method, namely mouth opening and closing judgment is carried out on the mouth coordinate information, and whether the driver is in a fatigue state is judged by detecting the yawning times of the driver in a period of time;

step S4: and (3) carrying out visual early warning on the judgment result of the fatigue behavior of the driver, which is obtained through the identification and analysis in the step (S2) and the step (S3).

3. The method for detecting fatigue driving of light YOLOv-pose according to claim 2, wherein the method comprises the steps of: in step S1, a AFLW dataset is introduced as a large-scale face dataset comprising multiple poses and multiple views, if the number of closed-eye pictures in the dataset is low enough to not meet the requirement, a CEW closed-eye dataset is introduced as a supplement, and when the format of the annotation file in the dataset is not the annotation format of the YOLO model, the annotation file is converted, the method is to process the data by using Python to generate a corresponding YOLO annotation file, and in step S1, the dataset is processed with 6:2:2, randomly sampling the proportion of the test data and dividing the test data into a training set, a verification set and a test set.

4. The method for detecting fatigue driving of light YOLOv-pose according to claim 2, wherein the method comprises the steps of: in step S2, lightweight YOLOv-pose introduces lightweight convolution GhostConv in the backbone network portion to use inexpensive linear transformation to generate a large number of Ghost feature maps capable of extracting required information from the original features at low cost, so as to reduce the number of parameters and calculation amount of the model;

in the step S2, a C3 module with high detection speed is introduced, two identical GhostConv modules are combined and connected through light weight operation on the C3 module to form a Ghost-Bottleneck structure, and the C3 module after recombination is used as a C3Ghost module, so that the complexity of a model is reduced, and the calculated amount of a face key point model is reduced.

5. The method for detecting fatigue driving of light YOLOv-pose according to claim 4, wherein the method comprises the steps of: in step S2, the lightweight YOLOv-pose uses a Slim-neg network structure as an enhanced feature fusion network, introduces lightweight convolution GSConv, furthest reserves implicit connection between each channel through dense convolution calculation to accelerate prediction calculation of a model, and constructs a GS-bottleneck structure through residual connection to further enhance the capability of network processing features; finally, the VOV-GSCSP module is formed by a one-time aggregation method, so that gradients at different positions can be mixed in a cross mode to enhance gradient expression and learning capability of the network.

6. The method for detecting fatigue driving of light YOLOv-pose according to claim 4, wherein the method comprises the steps of: in step S2, light YOLOv-pose is added to the occlusion awareness sea to emphasize the face region in the image and weaken the background while realizing multi-scale face detection, and improve the positioning effect of the facial key points.

7. The method for detecting fatigue driving of light YOLOv-pose according to claim 4, wherein the method comprises the steps of: in step S2, lightweight YOLOv-pose preserve the decoupling structure with a GNSC-Head detection Head and lightweight improvements are made to the network using group normalized shared convolution. .

8. The method for detecting fatigue driving of light YOLOv to pose according to claim 5, wherein: in step S2, a Pytorch deep learning framework is utilized to build a lightweight YOLOv-pose key point detection model, the number of training rounds is set to 300, the input image size is set to 640×640, the batch size is 64, training weights are stored once every 10 rounds, the verification set is utilized to calculate the mAP, the mAP performance is used to evaluate the model performance, and finally the model with the largest mAP is selected as the final model.

9. The method for detecting fatigue driving of light YOLOv-pose according to claim 2, wherein the method comprises the steps of: in step S3, according to the eye key point information output by the lightweight YOLOv-pose, calculating an eye aspect ratio EAR, wherein EAR threshold setting refers to the P80 standard in the PERCLOS standard, and P80 represents that the area of the eyelid covering the pupil exceeds 80% and is calculated as eye closure; the formula is as follows:

Wherein x _i represents the abscissa of the eye key points, y _i represents the ordinate, f _e is defined to judge the eye fatigue state of the driver, T _e is the number of eye closure frames in the detection time, and T _e is the total number of frames in the detection time;

In step S3, the aspect ratio MAR of the mouth is calculated based on the mouth keypoint information output from the lightweight YOLOv-pose. The formula is as follows:

Wherein x _i represents the abscissa of the key point of the mouth, y _i represents the ordinate, f _m is defined to judge the fatigue state of the mouth of the driver, T _m represents the number of frames of the mouth in the detection time, and T _m represents the total number of frames in the detection time.

10. The method for detecting fatigue driving of light YOLOv to pose according to claim 9, wherein the method comprises the steps of: in step S3, the output result of the model is evaluated by an algorithm, and whether the driver is in a fatigue state is judged by integrating various indexes, and a visual early warning is performed, wherein the triggering standard of the visual early warning is as follows:

Under normal state, the closing time of the monocular is 0.1-0.15 seconds, and under fatigue state, the closing time of the monocular is more than 0.5 seconds, namely if the closing frequency of the monocular is more than 0.5 in unit time, the driver is judged to be in fatigue state; typically the duration of the human yawning state is 3-5 seconds; in the algorithm in step S3, the unit detection time is selected to be 30 seconds, and the number of times of yawning in 30 seconds cannot exceed two times, that is, when the yawning frequency in the unit time is greater than 0.4, the driver is judged to be in a fatigue state.