CN112102413B - An automatic calibration method for vehicle-mounted cameras based on virtual lane lines - Google Patents

Abstract

The invention discloses a virtual lane line-based vehicle-mounted camera automatic calibration method, which comprises the following steps: a world coordinate system is established at the intersection point of the center of a rear axle of the vehicle and the ground vertically downwards, the Z axis is arranged right in front of the vehicle, the X axis is arranged on the right side of the advancing direction, and the Y axis is arranged vertically downwards; establishing a camera coordinate system; taking a single picture at the center of a lane in front of a vehicle by using a camera, measuring the lane width, selecting a rectangular frame formed by virtual lane lines on two sides as a calibration graph under the overlooking view angle of a world coordinate system, obtaining the relation between four characteristic points of the rectangle and the lane width according to the rectangular property, and obtaining a rotation matrix equation based on the camera coordinate system according to the orthogonal matrix property and the coordinate transformation relation between the camera coordinate system and the world coordinate system; the camera coordinates are converted into pixel coordinates by using the camera internal parameters, and then the pixel coordinates of the four characteristic points are acquired from the image, and then psi, theta, phi and h parameters of a rotation matrix and a translation matrix related to the camera external parameters are obtained.

Description

An automatic calibration method for vehicle-mounted cameras based on virtual lane lines

technical field

The invention belongs to the field of traffic, and in particular relates to an automatic calibration method for a vehicle-mounted camera based on virtual lane lines.

Background technique

So far, the automatic calibration algorithms in the traffic field (including vehicle cameras and traffic monitoring cameras, etc.) can be roughly divided into calibration algorithms based on static targets such as lane lines and calibration algorithms based on moving targets such as vehicles and pedestrians according to markers. Compared with the calibration algorithm based on static targets, the algorithm based on moving targets is much more complicated. It not only requires vehicles or pedestrians to appear in the picture, but also needs to analyze a video sequence to obtain the moving trajectory and then get the vanishing point. The algorithm even has requirements on the direction and speed of movement, so this method is more suitable for stationary traffic surveillance cameras. For vehicle-mounted cameras, there may be a large number of vehicles in the scene, but due to the complex relative motion between vehicles, it is difficult to find a suitable target for trajectory analysis, and lane lines, as stationary objects, are more suitable for use as vehicle-mounted cameras. Calibrated identifiers.

In the process of camera imaging, a point in the 3D world is transformed into a pixel in a 2D image. A geometric method can be used to build a model to describe the process. Camera parameters are some parameters involved in the geometric model. Camera internal parameters include focal length, optical center position, distortion coefficient, etc.; camera external parameters include rotation matrix and translation matrix. The purpose of camera calibration is to obtain camera parameters, and the calibration accuracy will directly affect the visual perception and positioning of autonomous vehicles. The traditional camera calibration method needs to use specific points on the calibration board to determine the camera parameters, so this method is only suitable for static conditions and is generally used to calibrate the internal parameters of the camera. While the vehicle-mounted camera is running, its external parameters may change due to various factors such as road bumps, vehicle body vibration, etc. (the internal parameters of the camera remain unchanged), and then the external parameters need to be recalibrated. Lane lines are ubiquitous in driving scenes, and the parallelism of lane lines and the known lane width can be used to automatically calibrate the camera extrinsic parameters.

Contents of the invention

The object of the present invention is to provide an automatic calibration method for a vehicle camera based on virtual lane lines to address the above-mentioned deficiencies in the prior art.

The present invention adopts following technical scheme to realize:

A method for automatic calibration of vehicle-mounted cameras based on virtual lane lines, comprising the following steps:

1) Establish a world coordinate system at the intersection of the center of the rear axle of the vehicle vertically downward and the ground. The front of the vehicle is the Z axis, the right side of the forward direction is the X axis, and the vertical downward is the Y axis; establish the camera coordinate system, the camera coordinate system The coordinates of the origin in the world coordinate system are (d, h, l);

2) Use the camera to take a single photo in the center of the lane in front of the vehicle, measure the width of the lane, and select the rectangular frame formed by the virtual lane lines on both sides as the calibration figure under the top view of the world coordinate system, and obtain four rectangles according to the nature of the rectangle. The relationship between feature points and lane width, and then the rotation matrix equation based on the camera coordinate system is obtained from the orthogonal matrix properties and the coordinate transformation relationship between the camera coordinate system and the world coordinate system;

3) Since the internal parameters of the camera remain unchanged during the driving process of the vehicle, the camera coordinates are converted into pixel coordinates by using the internal parameters of the camera, and then the pixel coordinates of the four feature points are obtained from the image, and the rotation matrix and translation matrix of the external parameters of the camera are obtained. ψ, θ, φ and h four parameters.

A further improvement of the present invention is that the specific implementation method of step 2) is as follows:

101) Introduce the transformation model between the world coordinate system W and the camera coordinate system C as follows

_Pc ＝R· _Pw +T

Where R represents the rotation matrix and T represents the translation matrix;

Since the rotation matrix R is an orthogonal matrix, the formula is rewritten as the following formula according to the nature of the orthogonal matrix:

P _w ＝R ^-1 P _c -R ^-1 T＝R ^T P _c -R ^T T

In the formula, the actual meaning of -R ^T T is the coordinates of the origin of the camera coordinate system in the world coordinate system;

In order to facilitate understanding and calculation, r _mn is used here to represent the elements in the rotation matrix R, and the formula is rewritten into a matrix form, as follows:

102) Let the four points of the rectangle be ABCD, A, C, B, and D are respectively on the same virtual lane line, distributed along the Y axis, and the width of the lane is width. According to the nature of the rectangle, the following formula is obtained:

103) Since the world coordinates are unknown, substitute (1) into (2), this process converts the world coordinates into camera coordinates, as follows

At this time, the world coordinates of each point are no longer included in the equation, only the camera coordinates of each point are left, and the internal parameters of the vehicle are unchanged during driving. Here, the internal parameters of the camera are considered to be known, so the camera coordinates are transformed into pixel coordinates.

A further improvement of the present invention is that the specific implementation method of step 3) is as follows:

201) Introduce the transformation model between the classic pixel coordinate system and the world coordinate system, as follows:

In the formula, f _x ＝f/dx; f _y ＝f/dy are called the normalized focal lengths of x-axis and y-axis respectively, dx and dy represent the physical size of a pixel in the direction of x-axis and y-axis respectively, and f is the focal length of the camera, (u ₀ , v ₀ ) represents the coordinates of the origin of the image coordinate system in the pixel coordinate system, R represents the camera rotation matrix, and T represents the camera translation matrix;

202) Combining the transformation model between the world coordinate system and the camera coordinate system with the transformation model between the pixel coordinate system and the world coordinate system, the transformation model of the camera coordinate system and the pixel coordinate system is obtained, as follows:

Expand to get the following formula:

f_x、f_y、u₀和v₀都为已知参数；In the formula,

f _x , f _y , u ₀ and v ₀ are all known parameters;

Substituting the above formula into the last equation in ①, the following formula can be obtained:

203) Solve the external parameter matrix R, T;

Substitute formulas ② and ③ into formula ① to eliminate the camera coordinates of each point, leaving only the parameters related to the pixel coordinates, and the pixel coordinates are directly obtained from the image; the equation is simplified to the following formula:

The formula contains only four unknowns ψ, θ, φ and h, so the following formula can be obtained by solving simultaneously:

In the formula, F _AC ＝(m _C -m _A )+tanφm _A m _C (n _A -n _C ); G _AC ＝sinφ(m _C -m _A )+cosφm _A m _C (n _A -n _C ); F _BD ＝(m _D -m _B )+tanφm _B m _D (n _B -n _D ); G _BD ＝sinφ(m _D -m _B )+cosφm _B m _D (n _B -n _D )

From this, the four parameters of the rotation matrix R of the camera extrinsic parameters and the translation matrix T of ψ, θ, φ and h are solved.

The present invention has at least the following beneficial technical effects:

The present invention provides a vehicle camera automatic calibration method based on virtual lane lines. The method uses the rectangles formed by the virtual lane lines on both sides of the image obtained by the vehicle camera in real time as the calibration object, and utilizes the parallelism of lane lines in common driving scenes. And the known lane width and other characteristics can automatically calibrate the camera extrinsic parameters to complete the camera calibration work at one time. Since the parameters of the selected lane line are known, and there are lane lines on common roads, the calibration method proposed by the present invention can realize real-time automatic Calibration has the advantages of simple operation, convenient measurement and good real-time performance.

Further, since the introduced unknown variable is only the width of the lane, and the coordinates of the four points of the selected virtual lane rectangle are transformed into coordinates in the camera coordinate system through the conversion relationship between the camera coordinate system and the world coordinate system, the present invention The selected calibration parameters are few, and the measurement is convenient. The coordinates of the four points obtained in the camera coordinate system are converted into the pixel coordinate system by the internal reference of the camera. At this time, the coordinates of the four points are converted into pixel coordinates in the pixel coordinate system of the image that can be directly obtained from the image. At this time, the camera The equations of the four parameters ψ, θ, φ, and h of the external parameter rotation matrix R and translation matrix T are converted into equations only about the width of the lane, and the external parameters of the camera can be obtained by solving them simultaneously. Therefore, the vehicle-mounted camera automatic calibration method based on virtual lane lines proposed by the present invention is simple to operate, uses few calibration parameters, is convenient to measure, has excellent universality and good real-time performance.

Description of drawings

Figure 1 is a schematic diagram from the world coordinate system to the camera coordinate system.

Fig. 2 is a schematic diagram of the rotation of the center point of the world coordinate system by an angle ψ around the X axis.

Fig. 3 is a schematic diagram from the camera coordinate system to the image coordinate system.

Fig. 4 is a schematic diagram of an image coordinate system to a pixel coordinate system.

Fig. 5 is a schematic diagram of the positional relationship between the vehicle and the camera, wherein Fig. 5(a) is a front view, Fig. 5(b) is a side view, and Fig. 5(c) is a top view.

FIG. 6 is a schematic diagram of a rectangle formed by virtual lane lines on both sides.

FIG. 7 is an image of the actual road scene used in the embodiment of the present invention after being calibrated by Opencv.

Fig. 8 is an image calibrated by the method of the present invention.

detailed description

The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

Camera Calibration Basic Theory

The process of taking images by a camera is an optical imaging process. The process involves the following four coordinate systems:

Pixel coordinate system: represented by (u,v), with the upper left corner of the image as the origin, the u axis horizontally to the right, and the v axis vertically downward, the unit is pixel.

Image coordinate system: represented by (x, y), the origin is the center of the image, and the horizontal to the right is the x-axis. Vertically downward is the y-axis, and the unit is physical unit.

Camera coordinate system: represented by (X _c , Y _c , Z _c ), the origin is the optical center of the lens, the X and Y axes are parallel to the two sides of the phase surface, and the Z axis is the optical axis of the lens, perpendicular to the image plane, and the unit is physical unit.

World coordinate system: represented by (X _w , Y _w , Z _w ), which describes the position of the camera. The position of the world coordinate system is not fixed and is defined by humans, and the unit is a physical unit.

World coordinate system to camera coordinate system

The transformation process from the world coordinate system to the camera coordinate system is a rigid body transformation, that is, the object does not deform during the transformation process, and only needs to perform rotation and translation operations on the coordinate system. The relationship between the world coordinate system and the camera coordinate system is shown in Figure 1, where R represents the rotation matrix and T represents the translation matrix.

Suppose there is a point P, the coordinates in the world coordinate system are P _w (X _w , Y _w , Z _w ), and the coordinates in the camera coordinate system are P _c (X _c , Y _c , Z _c ), then P _c and P _w has the following relationship:

P _c ＝R·P _w +T (5-1)

Since the camera coordinate system can be obtained by the rotation and translation of the world coordinate system, the present invention first rotates the point P around the X axis by an angle ψ, as shown in Figure 2:

According to the relationship between the two coordinate systems in Figure 2, the matrix form of the rotation angle ψ around the X axis in the world coordinate system can be obtained as shown in formula (5-2):

In the same way, it can be obtained that the coordinate change relationship after rotating around the Y axis by θ angle and around Z axis by φ angle is shown in formula (5-3).

Then the rotation matrix R is:

At this point, the relationship between the camera coordinate system and the world coordinate system can be obtained. Since the elements in the rotation matrix R are relatively long, in order to facilitate understanding and expression, they will be represented by subscripting r in the following text, as shown in formula (5-5):

Camera coordinate system to image coordinate system

This process is a process of transforming from a three-dimensional coordinate system to a two-dimensional plane coordinate system. The relationship between the two coordinate systems is a perspective projection, and it conforms to the triangle similarity theorem. The relationship between the two coordinate systems is shown in Figure 3, where f is the camera focal length.

It can be seen from the figure above that PO _c is the connection line between point P _c (X _c , Y _c , Z _c ) and optical center O _c , and the intersection point of PO _c and the imaging plane is the spatial point P _c (X _c , Y _c , Z _c ) projection point p(x, y) on the imaging plane, so the present invention can obtain two pairs of similar triangles ΔABO _c ~ ΔoCO _c , ΔPBO _c ~ ΔpCO _c , through the similarity of two pairs of similar triangles The relationship can be obtained as formula (5-6):

Rewrite the above formula into matrix form as follows:

Image coordinate system to pixel coordinate system

There is no rotation transformation in this transformation process, but the origin positions of the two coordinate systems are inconsistent, and the coordinate system unit sizes are also inconsistent, so it can be realized by scaling transformation and translation transformation. The relationship between the two coordinate systems is shown in Figure 4, (u ₀ , v ₀ ) represents the coordinates of the origin of the image coordinate system in the pixel coordinate system, and p(x,y) is the spatial point P _c (X _c ,Y _c , Z _c ) Projection points on the imaging plane.

Therefore, the relationship between the two coordinate systems can be expressed by the following formula:

In the formula, dx and dy represent the physical size of a pixel in the x and y axis directions, respectively. Then express the above formula with homogeneous coordinates and matrix as follows:

So far, the matrix relationships among the four coordinate systems have been obtained. After finishing the formulas (5-5), (5-7) and (5-9), the coordinate transformation relationship between the pixel coordinate system and the world coordinate system can be finally obtained, and its matrix form is shown in formula (5-10):

In the formula, f _x =f/dx; f _y =f/dy, which are respectively called the normalized focal lengths of the x-axis and the y-axis.

In formula (5-10), the first matrix after the second equal sign is the internal reference matrix of the camera, and the second matrix is the external reference matrix of the camera. Therefore, the camera internal parameters mainly include four parameters f _x , f _y , u ₀ and v ₀ and the distortion coefficient, which reflects the relationship between the camera coordinate system and the pixel coordinate system; the camera external parameters have 6 parameters in total, namely ψ, θ, φ and the three elements in the translation matrix T, the external parameters reflect the relationship between the world coordinate system and the camera coordinate system.

Automatic Calibration Method of Car Camera Based on Virtual Lane Lines

As shown in Figure 5, a world coordinate system and a camera coordinate system are established. The default camera coordinate system is the Z axis along the optical axis, the X axis to the right, and the Y axis vertically downward. The origin of the world coordinate system is at the intersection of the center of the rear axle of the vehicle vertically downward and the ground, the front of the vehicle is the Z axis, the right side of the forward direction is the X axis, and the vertical downward is the Y axis, the coordinates of the origin of the camera coordinate system in the world coordinate system is (d, h, l).

The relative position of the camera and the vehicle will change with the vibration of the vehicle while the vehicle is running. Generally speaking, the three rotation angles and the height of the camera in the external parameters of the camera change significantly, while d and l are basically unchanged. Therefore, the present invention proposes The automatic calibration algorithm mainly calculates the following four parameters: ψ, θ, φ and h. Assuming that the road surface is flat and the vehicle’s forward direction is parallel to the direction of the lane line, the present invention selects the rectangular frame formed by the virtual lane lines on both sides as the calibration figure, as shown in Figure 6:

Under the top view angle of the world coordinate system, the present invention considers that the four points ABCD form a rectangle, and formula (5-11) can be obtained according to the nature of the rectangle:

In the formula, width represents the lane width.

In formula (5-1), since the rotation matrix R is an orthogonal matrix, the formula can be rewritten as the following formula according to the nature of the orthogonal matrix:

P _w = R ^-1 P _c -R ^-1 T = R ^T P _c -R ^T T (5-12)

In the formula, the actual meaning of -R ^T T is the coordinates of the origin of the camera coordinate system in the world coordinate system.

Rewrite the formula (5-12) into a matrix form, as shown in the formula (5-13). For the convenience of understanding and calculation, r _mn is used here to represent the elements in the rotation matrix R.

Since the world coordinates are unknown, the present invention substitutes formula (5-13) into formula (5-11), and this process can convert the world coordinates into camera coordinates, as shown in formula (5-14).

At this point, the equation no longer contains the world coordinates of each point, only the camera coordinates of each point remain. It has been mentioned above that the internal parameters of the vehicle do not change during driving, and here the internal parameters of the camera are considered to be known. Therefore, the camera internal reference can be used to convert the camera coordinates into pixel coordinates.

From the formula (5-10), we can see that the relationship between the camera coordinate system and the pixel coordinate system is as follows:

Expand the formula (5-15) to get the following formula:

f_x、f_y、u₀和v₀都为已知参数，因此只需要再从图像中获取各点的像素坐标后即可计算出m、n。In the formula,

f _x , f _y , u ₀ and v ₀ are all known parameters, so m and n can be calculated only after obtaining the pixel coordinates of each point from the image.

Substituting formula (5-16) into the last equation in formula (5-14) gives:

Substituting formulas (5-16) and (5-17) into formula (5-14) can eliminate the camera coordinates of each point, leaving only parameters related to pixel coordinates, which can be obtained directly from the image. The equation simplifies to the following form:

Equation (5-18) only contains four unknowns ψ, θ, φ and h, so it can be solved simultaneously:

In the formula, F _AC ＝(m _C -m _A )+tanφm _A m _C (n _A -n _C ); G _AC ＝sinφ(m _C -m _A )+cosφm _A m _C (n _A -n _C ); F _BD =(m _D -m _B )+tanφm _B m _D (n _B -n _D ); G _BD =sinφ(m _D -m _B )+cosφm _B m _D (n _B -n _D ).

Fig. 7 is the original picture calibrated by Opencv (a computer vision and machine learning software library based on BSD licensing, open source distribution). After the vehicle is parked, the present invention actually measures the world coordinates of 8 points on the lane line. The function solvePnP can obtain the external parameters according to the world coordinates and the corresponding image coordinates; Fig. 8 is a picture calibrated using the calibration algorithm of the present invention, in which four vertices of a rectangle formed by virtual lane lines are selected. Table 1 is a comparison of calibration results and errors between a vehicle-mounted camera automatic calibration method based on virtual lane lines (the present invention) and the Opencv method in actual road scenes.

Table 1:

Claims (1)

1. an on-board camera automatic calibration method based on virtual lane lines, is characterized in that, comprises the following steps: 1) Establish a world coordinate system at the intersection of the center of the rear axle of the vehicle vertically downward and the ground. The front of the vehicle is the Z axis, the right side of the forward direction is the X axis, and the vertical downward is the Y axis; establish the camera coordinate system, the camera coordinate system The coordinates of the origin in the world coordinate system are (d, h, l); 2) Use the camera to take a single photo in the center of the lane in front of the vehicle, measure the width of the lane, and select the rectangular frame formed by the virtual lane lines on both sides as the calibration figure under the top view of the world coordinate system, and obtain four rectangles according to the nature of the rectangle. The relationship between feature points and lane width, and then the rotation matrix equation based on the camera coordinate system is obtained from the nature of the orthogonal matrix and the coordinate transformation relationship between the camera coordinate system and the world coordinate system; the specific implementation method is as follows: 101) Introduce the transformation model between the world coordinate system W and the camera coordinate system C as follows _Pc ＝R· _Pw +T Where R represents the rotation matrix and T represents the translation matrix; Since the rotation matrix R is an orthogonal matrix, the formula is rewritten as the following formula according to the nature of the orthogonal matrix: P _w ＝R ^-1 P _c -R ^-1 T＝R ^T P _c -R ^T T In the formula, the actual meaning of -R ^T T is the coordinates of the origin of the camera coordinate system in the world coordinate system; In order to facilitate understanding and calculation, r _mn is used here to represent the elements in the rotation matrix R, and the formula is rewritten into a matrix form, as follows:

102) Let the four points of the rectangle be ABCD, A, C, B, and D are respectively on the same virtual lane line, distributed along the Y axis, and the width of the lane is width. According to the nature of the rectangle, the following formula is obtained:

103) Since the world coordinates are unknown, substitute (1) into (2), this process converts the world coordinates into camera coordinates, as follows

At this time, the world coordinates of each point are no longer included in the equation, only the camera coordinates of each point are left, and the internal parameters of the vehicle are unchanged during driving. Here, the internal parameters of the camera are considered to be known, so the camera coordinates are transformed into pixel coordinates; 3) Since the internal parameters of the camera remain unchanged during the driving process of the vehicle, the camera coordinates are converted into pixel coordinates by using the internal parameters of the camera, and then the pixel coordinates of the four feature points are obtained from the image, and the rotation matrix and translation matrix of the external parameters of the camera are obtained. ψ, θ, φ and h four parameters; the specific implementation method is as follows: 201) Introduce the transformation model between the classic pixel coordinate system and the world coordinate system, as follows:

In the formula, f _x ＝f/dx; f _y ＝f/dy are called the normalized focal lengths of x-axis and y-axis respectively, dx and dy represent the physical size of a pixel in the direction of x-axis and y-axis respectively, and f is the focal length of the camera, (u ₀ , v ₀ ) represents the coordinates of the origin of the image coordinate system in the pixel coordinate system, R represents the camera rotation matrix, and T represents the camera translation matrix; 202) Combining the transformation model between the world coordinate system and the camera coordinate system with the transformation model between the pixel coordinate system and the world coordinate system, the transformation model of the camera coordinate system and the pixel coordinate system is obtained, as follows:

Expand to get the following formula:

In the formula,

f _x , f _y , u ₀ and v ₀ are all known parameters; Substituting the above formula into the last equation in ①, the following formula can be obtained:

203) Solve the external parameter matrix R, T; Substitute formulas ② and ③ into formula ① to eliminate the camera coordinates of each point, leaving only the parameters related to the pixel coordinates, and the pixel coordinates are directly obtained from the image; the equation is simplified to the following formula:

The formula contains only four unknowns ψ, θ, φ and h, so the following formula can be obtained by solving simultaneously:

In the formula, F _AC ＝(m _C -m _A )+tanφm _A m _C (n _A -n _C ); G _AC ＝sinφ(m _C -m _A )+cosφm _A m _C (n _A -n _C ); F _BD ＝(m _D -m _B )+tanφm _B m _D (n _B -n _D ); G _BD ＝sinφ(m _D -m _B )+cosφm _B m _D (n _B -n _D ) From this, the four parameters of the rotation matrix R of the camera extrinsic parameters and the translation matrix T of ψ, θ, φ, and h are solved.

Images