CN107167811A - The road drivable region detection method merged based on monocular vision with laser radar - Google Patents

Abstract

The invention discloses a road drivable area detection method based on the fusion of monocular vision and laser radar, which belongs to the field of intelligent transportation. Existing road detection methods for unmanned vehicles are mainly based on monocular vision, stereo vision, laser sensors, and multi-sensor fusion, which have disadvantages such as not being robust to illumination, complex 3D matching, sparse laser light, and low overall fusion efficiency. Although some supervised methods achieve better accuracy, the training process is complicated and the generalization effect is poor. The road drivable area detection method based on the fusion of monocular vision and laser radar proposed by the present invention uses superpixels and point cloud data fusion, uses features to make the machine self-learn the road area, and integrates the road obtained by Bayesian framework information to determine the final region. This method does not require strong hypothesis information and complex training process, and has superior generalization and robustness, extremely fast speed and high precision, and is easier to promote and use in practical applications.

Description

Road drivable area detection method based on fusion of monocular vision and lidar

technical field

The invention belongs to method research in the field of intelligent transportation, and relates to a road drivable area detection method based on the fusion of monocular vision and laser radar.

Background technique

In recent years, road detection has been an important part of research in the field of driverless driving. Currently widely used road detection methods are: monocular vision method, stereo vision method, lidar method and fusion-based method. Among them, the monocular vision method only considers the visual information of the scene, which is easily affected by lighting conditions and weather conditions; the stereo vision method consumes a lot of time in 3D reconstruction and is not suitable for practical applications; the lidar method has point cloud data Sparse disadvantage. The road detection method based on the fusion of pixel information and depth information not only makes full use of the texture, color and other information about the scene provided by the camera, but also combines the depth information of the lidar to make up for the shortcomings of visual information that are not robust to the environment. In terms of algorithm efficiency It overcomes the problems of low efficiency, difficulty in real-time calculation, and practical application of non-fusion methods. Therefore, the road detection method based on fusion has developed rapidly and become the first choice for road detection of unmanned vehicles. Fusion-based road detection method is an optimal road detection developed on the basis of monocular vision, lidar method, sensor fusion, etc. Therefore, it has been widely used in engineering practice, especially in unmanned vehicle driving.

Road detection for unmanned vehicles is also divided into supervised methods and unsupervised methods. Due to the diversity of road surface information, the complexity of scene information, and the variability of light and weather conditions, unmanned vehicles have high requirements for the robustness and generalization performance of road detection methods. Therefore, the unsupervised road detection method for unmanned vehicles is also an important part of the research in the field of unmanned driving. On the one hand, the unsupervised road detection method does not require a large amount of labeled data and time-consuming training process, and can learn road information independently according to the extracted features, which is a method with high generalization ability. On the other hand, the traffic scenes in the real world are complex and changeable. When it is impossible to provide training samples for all scenes for unmanned driving, supervised methods are dangerous when encountering driving scenes that are quite different from the training samples. The reliability is great, and the unsupervised road detection method is robust to almost all scenarios and is suitable for practical applications of unmanned driving.

Contents of the invention

The object of the present invention is to provide a road drivable area detection method based on the fusion of monocular vision and laser radar.

In order to achieve the above object, the present invention adopts the following technical solutions.

First of all, the fusion method uses the fusion of superpixel and laser point cloud data, and projects the point cloud data onto the superpixel segmented image according to the laser parameter calibration. The superpixel method makes full use of the texture characteristics of the scene and greatly reduces the positioning accuracy. The scope of the road area greatly improves the efficiency of the algorithm; secondly, use triangulation to find the spatial relationship of points, establish an undirected graph and calculate the normal vector of each point according to the obtained spatial relationship triangle, and classify obstacle points according to the undirected graph ; Then, this method defines a new feature (ray) based on the minimum filtering method, and finds the initial candidate area of the road area, further reduces the detection range of the road area, and greatly improves the algorithm efficiency; by defining a new feature (level ) quantifies the drivability of each point from the aspect of depth information, and effectively utilizes the depth information. In addition, the fusion method also uses an unsupervised fusion method, that is, based on the self-learning Bayesian framework, the probability of the candidate road area learned by fusing each feature (color feature, level feature, normal vector feature, intensity feature) Information, this algorithm has high efficiency and strong robustness.

The specific steps of the superpixel and laser point cloud data fusion are as follows:

Use the existing improved linear iterative clustering method combined with edge segmentation to perform superpixel segmentation on the pictures collected by the camera, and divide the picture into N superpixels, each superpixel p _c =(x _c ,y _c ,z _c , 1) ^T contains several pixels, where x _c , y _c , z _c represent the average value of the position information of all pixels in the superpixel in the camera coordinate system. At the same time, the RGB of these pixels are unified as The average RGB of all pixels in the superpixel. Then use the existing calibration technology to project each point p _l = (x _l ,y _l ,z _l ,1 ^T ) obtained by the lidar onto the image after superpixel segmentation, and finally obtain the point set Among them, P _i = ( _xi , y _i , _zi , u _i , v _i ), _xi , y _i , _zi represent the position information of the point in the laser coordinate system, (u _i , v _i ) represents The position information in the camera coordinate system corresponding to this point. Finally, through the co-point constraint, only the laser points projected near the edge of the superpixel are kept.

Define a new feature (ray) based on the minimum filtering method to find the initial candidate area of the road area; the specific steps are as follows:

First, define the initial candidate area for this road area as Among them, S _i represents is the set of all pixels contained in the superpixel S _i , define I _DRM as "direction ray map", and point set P _i = ( _xi , y _i , z _i , u _i , v _i ) _i , v _i ) coordinates are converted to polar coordinates with the center point (P _base ) of the last line of the picture as the origin, then is the set of points A proper subset of (u _i ,v _i ), where Indicates that the i-th point belongs to the h-th angle, means The collection of obstacle points in , the calculation method is shown in the following text and the flow chart.

Secondly, in order to solve the problem of laser ray leakage, the minimum filtering method is used to process the obtained I _DRM to obtain the expected I _DRM , and finally get

Define a new feature (level); the specific steps are as follows:

define each point The level feature is The algorithm is shown in the flow chart, and combined with the superpixel to obtain the level feature L(S _i ) of the superpixel S _i :

The specific steps for fusion using the self-learning Bayesian framework are as follows:

First, combined with the initial candidate area as The probabilities of candidate regions for unsupervised learning using four types of features respectively.

For the initial candidate region is The superpixel S _i in S _i , the RGB values of each pixel P _i = ( _xi , y _i , _zi , u _i , v _i ) contained in S i have been unified, using the Gaussian parameters μ _c and Self-learning color features, the formula is as follows:

θ=45°

Using Gaussian parameters μ _l and The formula of the level feature L(S _i ) of the self-learning superpixel S _i is:

Using Gaussian parameters μ _n and The formula of the normal vector feature N(S _i ) of the self-learning superpixel S _i is:

Define Sg(S _i ) as the number of rays passing through the superpixel S _i , and the formula for the intensity feature Sg(S _i ) of the self-learning superpixel S _i is:

Finally, the Bayesian framework is established to fuse the four features, the formula is as follows:

Among them, p(S _i =R|Obs) represents the probability that the superpixel S _i belongs to the road area, and Obs represents the observation based on these four features.

The beneficial effects of the present invention are reflected in:

First, since traditional fusion methods employ global fusion, this greatly limits the practicability and computational efficiency of these algorithms. The present invention proposes the use of fusion of superpixels with laser point cloud data. This method greatly reduces the range of candidates for the road area and greatly improves the efficiency of the algorithm. Second, therefore, the new feature (ray) proposed by the present invention finds the initial candidate area of the road area, further reduces the detection range of the road area, and greatly improves the algorithm efficiency. Third, the level feature proposed by the present invention digitizes the drivability of each point from the aspect of depth information, overcomes the sparseness of depth information, effectively utilizes depth information, and greatly contributes to the accuracy of the algorithm. Fourth, the fusion relationship between superpixels and depth information quantified by the intensity feature proposed by the present invention fully considers the problem that the visual information is near large and far small, and contributes greatly to the accuracy of the algorithm. Therefore, the algorithm has important research significance and extensive engineering application value. Fifth, the self-learning Bayesian framework integrates the probability information of the candidate road area learned by each feature. This algorithm has high efficiency and strong robustness.

Description of drawings

Figure 1 is a schematic block diagram of a road drivable area detection method based on monocular vision and lidar fusion;

Figure 2 is an algorithm flow chart for obtaining ray features;

Figure 3 is the effect diagram of the initial candidate area without using the minimum filter to process ray leakage (bottom) and after use (top);

Figure 4 is an algorithm flow chart for obtaining level features;

Figure 5 is an effect map of the probability distribution of the alternative road area obtained by self-learning of color features;

Figure 6 is an effect diagram of the probability distribution of the alternative road area obtained by level feature self-learning;

Figure 7 is an effect diagram of the probability distribution of the alternative road area obtained by the self-learning of the normal vector feature;

Figure 8 is an effect diagram of the probability distribution of the alternative road area obtained by self-learning of intensity features;

Fig. 9 is a probability distribution diagram of the final region obtained by the fusion of the self-learning Bayesian framework;

detailed description

Referring to Fig. 1, use the existing improved linear iterative clustering method combined with edge segmentation to perform superpixel segmentation on the picture collected by the camera, and divide the picture into N superpixels, each superpixel p _c =(x _c , y _c , z _c ,1) ^T contains several pixels, where x _c , y _c , z _c represent the average value of the position information of all pixels in the superpixel in the camera coordinate system. At the same time, these pixels The RGB of a point is unified as the average RGB of all pixels in the superpixel. Reuse existing calibration techniques and rotation matrices and transformation matrix According to the formula (1) to get the conversion matrix

Use the rotation matrix with Establish the conversion relationship between the two coordinate systems, such as formula (2):

Project each point p _l =(x _l ,y _l ,z _l ,1) ^T obtained by the lidar onto the image after superpixel segmentation, such as formula (3):

get point set Among them, P _i = ( _xi , y _i , _zi , u _i , v _i ), _xi , y _i , _zi represent the position information of the point in the laser coordinate system, (u _i , v _i ) represents The position information in the camera coordinate system corresponding to this point. Finally, only the laser spots near the edge of the superpixel are kept.

Using data fusion to classify obstacle points, the mapping relationship ob(P _i ) is obtained, ob(P _i )=1 means that P _i is an obstacle point, otherwise it is 0. For the coordinate system (u _i , v _i ) of P _i , use triangulation (Delaunay triangulation) to obtain many spatial triangles and generate undirected graphs E represents the set of edges in the graph that are associated with node P _i . Eliminate the edges (P _i , P _j ) whose Euclidean distance does not satisfy the formula (4) in the coordinate system (u _i , v _i ):

||P _i -P _j ||＜ε………………………………………………………(4)

Defined as the set of points connected to P _i as Nb(P _i ), then the surface of the space triangle related to and is {(u _j ,v _j )|j=iorP _j ∈ Nb(P _i )}. Calculate the normal vectors of each space triangle. Obviously, when the space _triangles around Pi are flatter and closer to the ground, the possibility of _Pi becoming a non-obstacle point is greater. We take the average value of the normal vectors of the space _triangles around Pi as P _i 's normal vector Formula (5) expresses the judging method of ob(P _i ):

Define the ray feature based on the minimum filter method and find the initial candidate area of the road area First, according to the algorithm flowchart shown in Figure 2, the "direction ray map"--I _DRM is obtained, wherein, Indicates that calculated according to the obstacle point classification method in the previous step The collection of obstacle points in , Indicates that P _i belongs to the hth angle. The algorithm transforms the (u _i , v _i ) coordinates of P _i = ( _xi , y _i , z _i , u _i , v _i ) into polar coordinates with the center point (P _base ) of the last line of the picture as the origin, then there is is the set of points Proper subset of (u _i , v _i ). Secondly, as shown in Figure 3, due to the sparsity of laser data, the problem of ray leakage needs to be dealt with. This method innovatively adopts the minimum filtering method to process the obtained I _DRM to obtain the expected I _DRM . Combined with superpixel segmentation, the initial candidate area for this road area is defined as Among them, S _i represents is the set of all pixels contained in the superpixel S _i , and finally fused with the superpixels to obtain

Define the level feature, Figure 4 gives the calculation of each point belonging to the h-th angle The level feature Formula (6) expresses that the level feature L(S _i ) of the superpixel S _i is obtained by combining with the superpixel:

As shown in Figure 5, the drivability probability information of the candidate road area obtained by using color features. For the initial candidate area The RGB value of the superpixel point S _i in S _i , each pixel point P _i = ( _xi , y _i , _zi , u _i , v _i ) contained in S i has been unified. The situation is not robust, so the color space conversion method is used to convert the original image I in the RBG space into the image I _log in the illuminant-invariant color space, such as formula (7):

Among them, I _log (u, v) is the pixel value under the I _log coordinate system (u, v), I _R , I _G , I _B represent the RGB value of I, and θ represents the constant angle perpendicular to the light change line . Equation (8) uses Gaussian parameters μ _c and Self-learning color features to obtain drivability probability information of candidate road areas:

As shown in Figure 6, the drivability probability information of the candidate road area obtained by using the level feature. Equation (9) uses Gaussian parameters μ _l and The level feature L(S _i ) of the self-learning superpixel S _i :

As shown in Figure 7, the drivability probability information of the candidate road area obtained by using the normal vector feature. calculate The normal vector feature N(S _i ) of the superpixel S _i in S i , that is, the height coordinate value of the point with the lowest normal vector in S _i Such as formula (10):

Equation (11) uses Gaussian parameters μ _n and The normal vector feature N(S _i ) of the self-learning superpixel S _i :

As shown in Figure 8, the drivability probability information of the candidate road area obtained by using the intensity feature. Sg(S _i ) is the number of rays passing through the superpixel S _i , and the intensity feature Sg(S _i ) of the self-learning superpixel S _i is shown in formula (12):

Finally, as shown in Figure 9, the Bayesian framework is established to fuse the four features to obtain the probability distribution map of the final area obtained by the fusion of the self-learning Bayesian framework, such as formula (13):

Among them, p(S _i =R|Obs) represents the probability that the superpixel S _i belongs to the road area, and Obs represents the observation based on these four features. It can be seen from Fig. 9 that this method completes the road detection task well.

In order to demonstrate the advantages of this method, we use datasets of 3 different environments on ROAD-KITTI benchmark, marked urban environment (Urban Marked, UM), multi-labeled urban environment (Urban Multiple Marked, UMM) and unlabeled Urban environment (Urban Unmarked, UU) test this method, from the maximum F-measure (Max F-measure, MaxF), average precision (Average Precision, AP), precision (Precision, PRE), recall rate (Recall, REC), The false positive rate (False Positive Rate, FPR) and the false negative rate (False Negative Rate, FNR) are analyzed. At the same time of the analysis, a comparative experiment was carried out, and compared with the currently published method MixedCRF and the fusion method RES3D-Velo which use laser to achieve the best results on the ROAD-KITTI benchmark data set, the comparison results are shown in Tables 1-4.

Table 1 is the comparative experimental results of this method (Ours Test), MixedCRF, and RES3D-Velo on the UM dataset:

Table 1 Comparative experimental results on the UM dataset

Table 2 is the comparative experimental results of this method (Ours Test), MixedCRF, and RES3D-Velo on the UMM dataset:

Table 2 Comparative experimental results on the UMM dataset

Table 3 is the comparative experimental results of this method (Ours Test), MixedCRF, and RES3D-Velo on the UU dataset:

Table 3 Comparative experimental results on the UU dataset.

Table 4 is the comparison result of the mean value of the experimental results of this method (Ours Test), MixedCRF, RES3D-Velo on the URBAN (ie UM, UMM, UU overall consideration) data set:

Table 4 Comparative experimental results in the URBAN dataset.

MixedCRF is a method that requires training. This method has obtained similar accuracy without any training, and has achieved the highest accuracy in the index of AP, which shows the superiority of this method.

In order to show the superiority of the self-learning Bayesian framework fusion adopted by this method, three different environment datasets were used on the ROAD-KITTI benchmark, the marked urban environment (Urban Marked, UM), the multi-labeled urban environment (Urban Marked, Multiple Marked, UMM) and unmarked urban environment (Urban Unmarked, UU) test this method, from the maximum F-measure (Max F-measure, MaxF), average precision (Average Precision, AP), precision (Precision, PRE) , Recall rate (Recall, REC), false positive rate (False Positive Rate, FPR), and false negative rate (False Negative Rate, FNR), these six indicators, analyze the initial candidate area (Initial) obtained by using a single ray feature , the accuracy of color feature (Color), intensity feature (Strength), level feature and normal vector feature (Normal), compared with the accuracy of Bayesian framework fusion (Fusion), the comparison results are shown in Table 5-8.

Table 5 is the initial candidate area (Initial), color feature (Color), intensity feature (Strength), level feature and normal vector feature (Normal) obtained by using ray feature alone, and Bayesian framework fusion (Fusion) in UM Comparison results on the dataset:

Table 5Comparison on UM Training Set(BEV).

Table 6 is the initial candidate area (Initial), color feature (Color), intensity feature (Strength), level feature and normal vector feature (Normal) obtained by using ray feature alone, and Bayesian framework fusion (Fusion) in UMM Comparison results on the dataset:

Table 6Comparison on UMM Training Set(Bev).

Table 7 is the initial candidate area (Initial), color feature (Color), intensity feature (Strength), level feature and normal vector feature (Normal) obtained by using ray feature alone, and Bayesian framework fusion (Fusion) in UU Comparison results on the dataset:

Table 7Comparison on UU Training Set(Bev).

Table 8 is the initial candidate area (Initial), color feature (Color), intensity feature (Strength), level feature and normal vector feature (Normal) obtained by using ray feature alone, and Bayesian framework fusion (Fusion) in UM , UMM, the comparison results of the average of the experimental results on the UU data set:

Table 8Comparison on URBAN Training Set(BEV).

It can be seen from Table 4 and Table 8 that the road drivable area detection method based on the fusion of monocular vision and lidar has achieved the highest accuracy AP. AP is also the most important indicator for measuring the detection method, and has also achieved good advantages in other indicators. , so this method is suitable for practical application.

The above content is a further detailed description of the present invention in conjunction with specific preferred embodiments. It cannot be determined that the specific embodiments of the present invention are limited thereto. Under the circumstances, some simple deduction or replacement can also be made, all of which should be regarded as belonging to the scope of patent protection determined by the submitted claims of the present invention.

Claims (5)

1. a kind of road drivable region detection method merged based on monocular vision with laser radar, it is characterised in that：

First, the method for the fusion merging using super-pixel and laser point cloud data, by cloud data according to laser parameter mark Surely project on the picture after super-pixel segmentation；

Secondly, the spatial relationship of point is found with triangulation, non-directed graph and simultaneously is set up according to the spatial relationship triangle drawn The normal vector of every bit is calculated, according to non-directed graph classification barrier point；

Then, the initial alternative area of road area is found using the method based on minimum filtering, road area is further reduced Detection range；By defining, new feature (level) quantizes the wheeled degree of each point in terms of depth information, in addition, Fusion method also utilizes a kind of unsupervised fusion method, the i.e. Bayesian frame based on self study, merges each feature, i.e. face Color characteristic, level features, normal direction measure feature, the probabilistic information for the alternative road area that strength characteristic learns.

2. the road drivable region detection method merged according to claim 1 based on monocular vision with laser radar, institute State super-pixel and merged with laser point cloud data and comprised the following steps that：

Clustered using linear iteraction and super-pixel segmentation is carried out to the picture that camera is gathered, be N number of super-pixel by picture segmentation, each Super-pixel p_c=(x_c,y_c,z_c,1)^TInclude several pixels, wherein x_c,y_c,z_cRepresent the phase of all pixels point in the super-pixel The average value of positional information under machine coordinate system, meanwhile, the RGB of these pixels unifies as all pixels point in the super-pixel Average RGB, recycle the calibration technique every bit p that obtains laser radar_l=(x_l,y_l,z_l,1)^TProject to super-pixel segmentation On picture afterwards, during laser is merged with point cloud, it is proposed that concurrent is constrained, concurrent constraint refers to, in super-pixel and laser point cloud data The laser spots of super-pixel adjacent edges are only retained in during fusion, point set is finally givenWherein P_i=(x_i, y_i,z_i,u_i,v_i), x_i,y_i,z_iRepresent positional information under the laser coordinate system of the point, (u_i,v_i) represent the corresponding phase of point Positional information under machine coordinate system.

3. the road drivable region detection method merged according to claim 1 based on monocular vision with laser radar, its It is characterised by：Method definition based on minimum filtering, finds the initial alternative area of road area；Comprise the following steps that：

First, the initial alternative area for defining the road area isWherein, S_iRepresentIt is Super-pixel S_iComprising all pixels point set, define I_DRMFor " oriented radial figure (direction ray map) ", and By point set P_i=(x_i,y_i,z_i,u_i,v_i) (u_i,v_i) coordinate transformation is to the central point (P of picture bottommost a line_base) it is original Under the polar coordinates of point, then haveIt is point set(u_i,v_i) proper subclass, wherein Table Show that belong to h angle at i-th point,Represent beIn barrier point set, Calculating process is as follows：

1) I is initialized_DRMIt is size and to be originally inputted picture identical full 0 matrix；

2) for h angle set a littleFind obstacle point set therein

If 3)Construct containerOtherwise,

4) willIt is included into I_DRM；

If 5) h=H, terminate；Otherwise, 2 are returned；

The problem of next is " ray leakage ", carries out subsequent treatment using the method for minimum filtering, obtains desired I_DRM, finally Obtain

4. the road drivable region detection method merged according to claim 1 based on monocular vision with laser radar, its It is characterised by：Define new feature (level)；The wheeled degree of level character representation corresponding points, calculating process is as follows：

1) initialization h angle set a littleThe level features of middle every bit

2) forIn i-th point, if

If 3) i≤N^(h), return to 2；

If 4) h=H, terminate；Otherwise, 1 is returned；

And combined with super-pixel and obtain super-pixel S_iLevel feature L (S_i)：

。

5. the road drivable region detection method merged according to claim 1 based on monocular vision with laser radar, its It is characterised by：Merged using self study Bayesian frame；Comprise the following steps that：

Self study -- color characteristic is carried out using feature in four, level features, normal direction measure feature, strength characteristic, first, with reference to Initially alternative area isFour kinds of feature unsupervised learnings of the person of being utilized respectively alternative area probability；

It is for initial alternative areaIn super-pixel point S_i, S_iComprising each pixel P_i=(x_i,y_i,z_i,u_i,v_i) Rgb value unified, utilize Gaussian parameter μ_cWithSelf study color characteristic, formula is as follows：

Utilize Gaussian parameter μ_lWithSelf study super-pixel S_iLevel feature L (S_i) formula be：

Utilize Gaussian parameter μ_nWithSelf study super-pixel S_iNormal direction measure feature N (S_i) formula be：

Define Sg (S_i) it is through super-pixel S_iRay quantity, self study super-pixel S_iStrength characteristic Sg (S_i) formula be：

Finally, set up Bayesian frame and merge 4 kinds of features, formula is as follows：

Wherein, p (S_i=R | Obs) represent super-pixel S_iBelong to the probability of road area, Obs represents the sight based on these four features Survey.