CN117612142B - Head posture and fatigue state detection method based on multi-task joint model - Google Patents

Abstract

The invention discloses a head posture and fatigue state detection method based on a multi-task joint model, which comprises the following steps of: designing an enhanced feature extraction network based on an aggregation and diversion mechanism on the basis of YOLOv; adding a head posture estimation branch fused with a large kernel attention mechanism in the model; labeling the face data set to form a fatigue driving data set; training a fatigue distraction detection model through a target detection loss function and a head posture estimation loss function; deploying the model on the vehicle-mounted terminal equipment, detecting the head gesture and the fatigue state through the model, and outputting information; whether in a tired state or a distraction state is determined by comparing a certain category duration with a set threshold value. The invention improves the generalization performance, robustness, reliability and detection precision of the model, reduces the training time and calculation resources of the model, improves the safety of a driver, reduces the fatigue distraction behavior in driving and reduces the occurrence rate of traffic accidents.

Description

Head posture and fatigue state detection method based on multi-task joint model

Technical Field

The present invention relates to the field of computer vision technology, and in particular to a head posture and fatigue state detection method based on a multi-task joint model.

Background technique

With the rapid development of the automobile industry, driving safety has received increasing attention. Fatigue driving, as a key safety hazard, has attracted widespread attention. Fatigue driving may cause drivers to react slowly and make misjudgments, thereby increasing the risk of traffic accidents. In order to improve road traffic safety, real-time monitoring and early warning of driver fatigue are crucial.

Traditional methods of fatigue driving detection are mainly based on the analysis of the driver's physiological signals, such as EEG and heart rate, but these methods require contact with the driver's body, and the installation and use process is relatively complicated. In recent years, fatigue driving detection technology based on computer vision has developed rapidly. Such methods mainly judge whether the driver is fatigued by analyzing the driver's facial features, such as the state of the eyes and mouth. However, these methods often rely on high-computing power equipment for real-time analysis, which not only consumes a lot of computing resources, but also may have a high false alarm rate.

As more and more scenarios require processing multiple tasks or goals at the same time, multi-task joint models have emerged. A multi-task joint model is a machine learning model that can process multiple tasks at the same time, and these tasks can be of different types or in different fields; its basic idea is to connect different tasks by sharing underlying features such as convolutional layers or word embedding layers.

However, there may be some disadvantages when using the multi-task joint model. For example, the correlation between different tasks may not be exactly the same, resulting in some tasks not being fully learned during the training process; there may be redundancy and noise in the information between different tasks, resulting in the model being disturbed during the training process; the number of parameters of the multi-task joint model may increase, resulting in the model being plagued by overfitting during the training process, etc.

Therefore, in the fatigue driving detection method based on the multi-task joint model, how to ensure the utilization of the correlation between different tasks, improve the generalization performance of the fatigue detection model, enhance the robustness and reliability of the model, reduce the time and computing resources of model training, while properly handling the relationship and contradiction between different tasks, the consumption of data resources and computing resources, the scalability and maintainability of the model, and the training and optimization of the model, etc., are the hot spots and difficulties in the current research on fatigue driving detection methods.

Summary of the invention

The purpose of the present invention is to provide a head posture and fatigue state detection method based on a multi-task joint model, which can improve the generalization performance of the model, enhance the robustness and reliability of the model, reduce the time and computing resources of model training, and properly handle the relationship and contradiction between different tasks. It can be used in various types of automobile driving scenarios, improve the safety of drivers, reduce fatigue and distraction behaviors during driving, and reduce the incidence of traffic accidents.

To achieve the above object, the present invention provides a head posture and fatigue state detection method based on a multi-task joint model, comprising the following steps:

S1: Design of fatigue and distraction detection model, taking YOLOv6 as the baseline model for improvement, and designing an enhanced feature extraction network based on aggregation and diversion mechanism on the basis of YOLOv6;

S2: Add a head posture estimation branch that integrates the large core attention mechanism to the fatigue distraction detection model;

S3: Prepare a face dataset and annotate the face dataset to form a fatigue driving dataset; when annotating, in addition to annotating the category and detection frame of each target, add an additional label to the face data to indicate whether the head rotation angle is greater than 45°; the annotated categories include eyes open, eyes closed, mouth open, and mouth closed;

S4: training the fatigue distraction detection model through the target detection loss function and the head posture estimation loss function;

S5: deploying the fatigue and distraction detection model on the vehicle-mounted terminal device, inputting the video stream captured by the terminal device camera into the fatigue and distraction detection model, detecting the head posture and fatigue state through the trained fatigue and distraction detection model and outputting information, wherein the output information includes the target category, the detection frame and whether the head rotation angle is greater than 45°;

S6: Determine whether the user is in a fatigue or distracted state by comparing the duration of a certain category with a set threshold.

Furthermore, in step S1, the enhanced feature extraction network based on the aggregation and diversion mechanism includes using a low-level aggregation and diversion mechanism to replace the upsampling fusion stage of the enhanced feature extraction network in YOLOv6, and using a high-level aggregation and diversion mechanism to replace the downsampling fusion stage of the enhanced feature extraction network in YOLOv6.

Furthermore, the aggregation and diversion mechanism includes an information alignment module, an information fusion module and an information diversion module; the information alignment module collects multi-layer feature maps from the backbone network and aligns them by upsampling or downsampling; the information fusion module fuses the aligned features to generate global features; the information diversion module uses a self-attention mechanism to divert global features to each feature layer.

Furthermore, in step S2, the head posture estimation branch integrating the large core attention mechanism is composed of multiple convolutional layers, a large core attention mechanism module and a fully connected layer.

Furthermore, the large-core attention mechanism module is capable of capturing long-distance relationships; the large-core attention mechanism module uses a large-core convolution layer to establish global correlations and produce attention results, while using depthwise separable convolution to reduce the amount of parameters.

Furthermore, in step S4, the target detection loss function and the head posture estimation loss function are composed of two parts, namely, a regression loss function based on SIoU and a classification loss function based on the classification and regression alignment method; the head posture estimation loss function is a cross entropy loss function between the model prediction result and the true label value, and the two losses are balanced by weight parameters to perform model training.

Furthermore, the fatigue driving dataset obtained in step S3 is divided into training set, validation set and test set in a ratio of 8:1:1. When loading the dataset in the training stage, the masoic and mixup data enhancement methods are used to improve the data robustness, and the sample size of the type with less data volume is increased through the data enhancement methods of horizontal and vertical flipping, random rotation, random cropping, deformation and scaling.

Furthermore, the fatigue and distraction detection model is obtained by training a convolutional neural network, which includes a backbone network, an aggregation and diversion enhanced feature extraction network, a target detection head, and a large core attention mechanism head posture estimation branch; the backbone network is used to extract image features; the target detection head outputs a detection box and category; the target detection head includes a classification regression branch, a bounding box regression branch, and a depth information regression branch; the large core attention mechanism head posture estimation branch outputs whether the head is turned.

Beneficial effects of the present invention:

The invention discloses a head posture and fatigue state detection method based on a multi-task joint model, designs an enhanced feature extraction network based on an aggregation and shunting mechanism, collects and fuses feature information of different scales through a unified module, and then shunts the fused features to different layers, thereby avoiding the problem of information loss inherent in the enhanced feature extraction network structure in YOLOv6, and enhancing the feature information fusion capability of the feature extraction network part without significantly increasing the reasoning time; and the aggregation and shunting mechanism strengthens the global feature extraction capability of the model and the learning of the global information of the picture, thereby improving the detection capability of the model.

The present invention adds a head posture estimation branch that integrates a large core attention mechanism to further strengthen the learning of global information of the image, increase the model's ability to learn head posture, and thus improve the accuracy of head posture estimation; this branch simplifies the regression problem into a classification problem, which has the advantages of good real-time performance and high accuracy. In addition, this branch does not require cumbersome key point annotation during training, which facilitates fine-tuning in different scenes and tasks, and reduces the impact of camera position on detection results to a certain extent, thereby improving the robustness of the model.

The head posture estimation branch of the present invention shares weights with the fatigue state detection branch. The former locates the positions of the eyes and mouth, providing additional semantic information to the latter, so that the model has a better effect on this task; the large-core attention mechanism module of the head posture estimation branch can capture long-distance relationships, thereby effectively extracting global features of the face and performing head posture estimation. The large-core attention mechanism module uses a large-core convolutional layer to establish global correlation and generate attention results, and uses deep separable convolution to reduce the number of parameters and reduce the model inference time.

BRIEF DESCRIPTION OF THE DRAWINGS

Fig. 1 is a diagram showing the working principle of the present invention.

Figure 2 is a schematic diagram of CUDA heterogeneous parallel computing.

Detailed ways

The present invention is further described in detail below with reference to the accompanying drawings.

1 and 2, a head posture and fatigue state detection method based on a multi-task joint model includes the following steps:

S1: The design of the fatigue and distraction detection model is improved based on YOLOv6 as the baseline model. On the basis of YOLOv6, an enhanced feature extraction network based on the aggregation and diversion mechanism is designed.

The enhanced feature extraction network based on the aggregation and diversion mechanism uses a low-level aggregation and diversion mechanism to replace the upsampling fusion stage of the enhanced feature extraction network in YOLOv6, and uses a high-level aggregation and diversion mechanism to metabolize the downsampling fusion stage of the enhanced feature extraction network in YOLOv6; feature information of different scales is collected and fused through a unified module, and then the fused features are diverted to different layers, which not only avoids the problem of information loss inherent in the enhanced feature extraction network structure in YOLOv6, but also enhances the feature information fusion capability of the feature extraction network part without significantly increasing the reasoning time.

The aggregation and diversion mechanism includes an information alignment module, an information fusion module, and an information diversion module. Among them, the information alignment module collects multi-layer feature maps from the backbone network and aligns them by upsampling or downsampling. The information fusion module fuses the aligned features to generate global features. The information diversion module uses the self-attention mechanism to divert global features to each feature layer. The aggregation and diversion mechanism can effectively divert the fused global information to each feature layer, thereby enhancing the global feature extraction capability of the model. Therefore, the enhanced feature extraction network based on the aggregation and diversion mechanism strengthens the model's learning of the global information of the image and improves the model's detection capability.

S2: Add a head pose estimation branch that integrates the large core attention mechanism to further strengthen the learning of the global information of the image, increase the ability of learning head pose for the model, and thus improve the accuracy of head pose estimation. The head pose estimation branch that integrates the large core attention mechanism consists of multiple convolutional layers, a large core attention mechanism module, and a fully connected layer. This branch simplifies the regression problem into a classification problem, and directly determines whether the head rotation angle is greater than 45°. It has the advantages of good real-time performance and high accuracy. In view of its sensitivity to occlusion and noise and low accuracy of head pose estimation in natural scenes, different scenes and face data sets are added to train the updated model when training the fatigue distraction detection model.

This branch does not require tedious key point annotation during training, which facilitates fine-tuning in different scenarios and tasks, and reduces the impact of camera position on detection results to a certain extent, thereby improving the robustness of the model. At the same time, the head pose estimation branch shares weights with the fatigue state detection branch. The former locates the position of the eyes and mouth, providing additional semantic information to the latter, so that the model has better results on this task. And the large-core attention mechanism module can capture long-distance relationships, thereby effectively extracting global features of the face and performing head pose estimation. The large-core attention mechanism module uses large-core convolutional layers to establish global correlations and produce attention results, while using deep separable convolutions to reduce the number of parameters and reduce model inference time.

S3: Prepare a face data set, and annotate the face data set to form a fatigue driving data set; in this embodiment, video data is collected by a camera, and a fatigue driving data set is formed after manual annotation. The convolutional neural network is trained using the fatigue driving data set to obtain a fatigue distraction detection model. When annotating, in addition to annotating each target category and detection frame, an additional label is added to the face data to indicate whether the head rotation angle is greater than 45°; the annotated categories include eyes open, eyes closed, mouth open, and mouth closed.

The obtained fatigue driving dataset is divided into training set, validation set and test set in the ratio of 8:1:1. When loading the dataset in the training stage, the masoic and mixup data augmentation methods are used to improve the data robustness, and the sample size of the type with less data volume is increased through horizontal and vertical flipping, random rotation, random cropping, deformation and scaling data augmentation methods to improve the generalization ability of the model.

The convolutional neural network includes a backbone network, an aggregation and diversion enhanced feature extraction network, a target detection head, and a large core attention mechanism head posture estimation branch; the backbone network is used to extract image features, and the target detection head outputs detection boxes and categories; the target detection head includes a classification regression branch, a bounding box regression branch, and a depth information regression branch. The classification regression branch and the bounding box regression branch output categories and detection boxes respectively, and the large core attention mechanism head posture estimation branch outputs whether the head is turned; the depth information regression branch is set to distinguish between people in the front and rear seats in the car to avoid misidentification.

The depth information regression branch consists of multiple convolutional layers, a pooling layer, and a fully connected layer. The decoding method of the depth information is as follows: first, the depth information inside the car is evenly divided into s stages, that is, for the depth span of [0, V], each depth span is V/s, and the representative depth of the segment is u=V/s. Then, for a classification model of s classes, the product of the probability of each class and the representative depth of the current class is taken as the final prediction value. Since the YOLOv6 model predicts based on anchor points, that is, the bounding box information and category information are predicted for each anchor point, the coarse-grained depth estimation is also based on anchor points, that is, the bounding box information, category information, and depth information are predicted for each anchor point.

The coarse-grained depth value interval decoded through the depth information backhaul is used to distinguish the front-row occupants and the back-row occupants. When the coarse-grained depth value is in the interval [0,1], it is the front-row occupants, and when the coarse-grained depth value is in the interval [1,2], it is the back-row occupants. The coarse-grained depth value refers to the feature scaling of the distance from the target object recognized and detected by the fatigue and distraction detection model to the camera. The actual distance is coarse-grainedly scaled to the interval [0,2]. If it is in the interval [0,1], it means that the target object is closer to the camera and is identified as being in the front row of the car. If it is in the interval [1,2], it means that the target object is farther from the camera and is identified as being in the back row of the car, thereby avoiding misidentification.

S4: The fatigue distraction detection model is trained through the target detection loss function and the head posture estimation loss function. The target detection loss function and the head posture estimation loss function are respectively the regression loss function based on SIoU and the classification loss function based on the classification and regression alignment method. The head posture estimation loss function is the cross entropy loss function between the model prediction result and the true label value; and the two losses are balanced through the weight parameter to perform model training.

S5: deploy the fatigue and distraction detection model on the vehicle-mounted terminal device, input the video stream captured by the terminal device camera into the fatigue and distraction detection model, and output detection information; the output detection information includes the target category, detection frame and whether the head rotation angle is greater than 45°. The deployment method of the fatigue and distraction detection model includes the following steps:

First, convert the fatigue and distraction detection model into an ONNX model, and then convert ONNX into a TensorRT model. Specifically, first convert the trained network into an ONNX model using the Pytorch internal interface, use the parser in TensorRT to read the ONNX model and build the engine; then call the TensorRT C++ interface and the Libtorch library to implement the model post-processing part. During the inference process, attention should be paid to the allocation of video memory. During calculation, use the CUDA library to move data from the CPU to the GPU, and then move data from the GPU back to the CPU after the inference calculation.

S6: Determine whether the driver is in a fatigued or distracted state by whether the duration of a certain category exceeds the set threshold. Determine whether the driver is looking around or driving distracted based on the changes in head posture and eye and mouth position; determine whether the driver is driving fatigued based on head posture information, eye opening and closing, and mouth opening and closing. The relevant detection categories for judging whether the driver is driving fatigued by eyes and mouth are divided into open eyes, closed mouth, closed eyes and open mouth. Because the characteristics of a person in a fatigued state are intuitive and obvious, such as the number of blinks, eye movement, yawning, nodding, etc., these states will be recorded by the camera and identified and judged.

The specific judgment method flow is as follows: first take a frame of image for face detection. If it is a face, locate the mouth and eyes and extract fatigue information and head turning information, and then perform information fusion; if it is a set abnormal state, accumulate until the duration is greater than the threshold. In this embodiment, the threshold is set to 3s, then it is determined to be a fatigue state, and a warning or prompt is given; if the acquired image is recognized as a non-face, randomly take another frame of image and repeat the above process. The fatigue state of the human eye is a state within a time period, so the PERCLOS method is used to determine that when the eye opening is greater than 20%, it is determined to be open, and less than or equal to 20% is considered to be closed. For mouth state detection, because there are many states of the mouth, among which yawning is a manifestation of fatigue, as long as the yawning mouth state is distinguished from other states, it is possible to determine whether the driver is fatigued. The opening degree of the mouth is calculated by the geometric shape of the mouth, the position of the mouth is marked with a rectangular frame, and the opening degree of the mouth is calculated by the ratio of the height to the width of the mouth. When the mouth opening degree is greater than 0.8, it is judged as an open mouth state of yawning, and when the mouth opening degree is less than or equal to 0.8, it is judged as a closed mouth state.

Claims (8)

1. A head posture and fatigue state detection method based on a multi-task joint model is characterized by comprising the following steps of: the method comprises the following steps:

S1, designing a fatigue distraction detection model, taking YOLOv as a baseline model for improvement, and designing a reinforced feature extraction network based on an aggregation and diversion mechanism on the basis of YOLOv;

s2, adding a head gesture estimation branch fused with a large nuclear attention mechanism into the fatigue distraction detection model;

s3, preparing a face data set, and labeling the face data set to form a fatigue driving data set; when in labeling, except labeling the category and the detection frame of each target, and adding an additional label for judging whether the head rotation angle is larger than 45 degrees to the face data; the marked categories comprise eye opening, eye closing, mouth opening and mouth closing;

S4, training a fatigue distraction detection model through a target detection loss function and a head posture estimation loss function;

S5, deploying a fatigue distraction detection model on the vehicle-mounted terminal equipment, inputting a video stream shot by a camera of the terminal equipment into the fatigue distraction detection model, detecting the head gesture and the fatigue state through the trained fatigue distraction detection model, and outputting information, wherein the output information comprises the type of a target, a detection frame and whether the head rotation angle is larger than 45 degrees;

And S6, comparing the duration of a certain category with a set threshold value to judge whether the vehicle is in a fatigue state or a distraction state.

2. The method for detecting head gestures and fatigue states based on the multi-task joint model according to claim 1, wherein the method comprises the following steps: in the step S1, the enhanced feature extraction network based on the clustering and splitting mechanism includes using a low-level clustering and splitting mechanism to replace the upsampling fusion stage of the enhanced feature extraction network in YOLOv, and using a high-level clustering and splitting mechanism to replace the downsampling fusion stage of the enhanced feature extraction network in YOLOv.

3. The method for detecting the head posture and the fatigue state based on the multi-task joint model according to claim 2, wherein the method comprises the following steps: the aggregation and distribution mechanism comprises an information alignment module, an information fusion module and an information distribution module; the information alignment module collects multi-layer feature graphs from a backbone network and performs alignment in an up-sampling or down-sampling mode; the information fusion module fuses the aligned features to generate features in a global scope; the information splitting module splits global features to the feature layers using a self-attention mechanism.

4. A method for detecting head pose and fatigue state based on a multi-task joint model according to any of claims 1-3, wherein: in the step S2, the head pose estimation branch fused with the large-core attention mechanism is composed of a plurality of convolution layers, a large-core attention mechanism module and a full connection layer.

5. The method for detecting the head posture and the fatigue state based on the multi-task joint model according to claim 4, wherein the method comprises the following steps: the large-core attention mechanism module can capture long-distance relations; the large kernel attention mechanism module uses large kernel convolution layers to build global dependencies and produce attention results while reducing the number of parameters using depth separable convolutions.

6. The method for detecting head gestures and fatigue states based on the multi-task joint model according to claim 1, wherein the method comprises the following steps: in the step S4, the target detection loss function and the head pose estimation loss function are composed of two parts, namely a SIoU-based regression loss function and a classification loss function based on a classification and regression alignment method; the head pose estimation loss function is a cross entropy loss function of a model prediction result and a real label value, and the model is trained by balancing the two losses through weight parameters.

7. The method for detecting head gestures and fatigue states based on the multi-task joint model according to claim 1, wherein the method comprises the following steps: dividing the fatigue driving data set obtained in the step S3 into a training set, a verification set and a test set according to the proportion of 8:1:1, improving the data robustness by using masoic and mixup data enhancement methods when the data set is loaded in the training stage, and increasing the sample size of the type with smaller data size by using the data enhancement modes of horizontal and vertical overturn, random rotation, random cutting, deformation and scaling.

8. The method for detecting head gestures and fatigue states based on the multi-task joint model according to claim 1, wherein the method comprises the following steps: the fatigue distraction detection model is trained by a convolutional neural network, and the convolutional neural network comprises a backbone network, an aggregation and shunt reinforcement feature extraction network, a target detection head and a head gesture estimation branch of a large-core attention mechanism; the backbone network is used for extracting picture characteristics; the target detection head outputs a detection frame and a category; the target detection head comprises a classification regression branch, a boundary box regression branch and a depth information regression branch; the head gesture of the large-core attention mechanism estimates whether the branch outputs a turning result or not.