CN116258817A - A method and system for constructing an autonomous driving digital twin scene based on multi-view 3D reconstruction - Google Patents

Abstract

The invention provides an automatic driving digital twin scene construction method and system based on multi-view three-dimensional reconstruction. The invention firstly provides an automatic driving digital twin scene construction system based on multi-view three-dimensional reconstruction, which comprises six modules including data acquisition and processing, camera pose estimation, multi-view three-dimensional reconstruction, point cloud model scale correction, empty point cloud model fusion and model precision quantitative evaluation. The multi-view three-dimensional reconstruction method of the automatic driving scene is also provided, and the problems of insufficient feature extraction in the texture complex region, large influence by noise data and the like in the prior art are solved, so that the reconstruction precision of the three-dimensional model of the automatic driving scene is improved, and the occupied space of a video memory is reduced. In addition, the invention also provides an automatic outside-cab scene image acquisition scheme based on the unmanned aerial vehicle, which can effectively and comprehensively acquire image data required by the multi-view three-dimensional reconstruction method.

Description

A self-driving digital twin scene construction method based on multi-view 3D reconstruction and system

technical field

The invention relates to the field of computer vision and the field of automatic driving technology, in particular to a method and system for constructing a digital twin scene for automatic driving based on multi-view three-dimensional reconstruction.

Background technique

In recent years, limited by the perception range and capabilities of a single vehicle, autonomous driving has gradually developed towards the direction of intelligent networking with digital twins as the core. The autonomous driving digital twin technology builds high-precision traffic scenes in a three-dimensional virtual environment based on sensor data in real scenes. The reconstructed 3D virtual scene not only provides a complete, sufficient, and editable scene for the simulation test of automatic driving, but also provides a large amount of driving data for the optimization of automatic driving algorithms, and provides an efficient solution for the automatic production of more high-precision maps. The autonomous driving digital twin technology establishes the connection between the physical world and the virtual world by mapping the elements of the real traffic scene to the virtual space. In the virtual world, people can control all aspects of autonomous driving in the whole process and all elements, and freely switch time and space according to certain logic and rules, research key technologies in autonomous driving at low cost and high efficiency, and then drive the real world development of autonomous driving technology.

The essence of 3D virtual scene reconstruction for autonomous driving is the perception and restoration of the 3D geometric structure of the traffic scene. Compared with the lidar scanning scheme, the vision-based method analyzes the multi-view geometric relationship and internal relationship from the two-dimensional projection, which has the advantages of low cost, convenient data acquisition, dense reconstruction model, and rich texture. The traditional Colmap multi-view 3D reconstruction algorithm is based on classical computational geometry theory, uses the normalized cross-correlation matching method to measure the optical consistency between views, and uses PatchMatch for depth transfer. The Colmap algorithm is highly interpretable, but it takes a long time to calculate the depth map, making it difficult to apply it to large-scale outdoor scenes such as autonomous driving. In recent years, with the development of deep learning, deep neural networks have shown a strong ability to extract image features. Many studies feed multiple views and corresponding camera poses into deep neural networks to achieve end-to-end multi-view stereo matching and depth map estimation. MVSNet proposes a differentiable homography to construct a cost body, and uses a multi-scale Unet-like structure to perform three-dimensional convolution on the aggregated cost body to obtain a smoothed probability body, and estimates the depth map of each image accordingly. CVP-MVSNet optimizes the quality of the depth map from coarse to fine by designing a pyramid structure; Fast-MVSNet uses a sparse cost body construction method to improve the speed of depth map estimation. Compared with the traditional Colmap algorithm, these algorithms greatly reduce the depth map estimation time.

Although the existing technology has achieved good results on indoor scene datasets, it faces great challenges in large-scale scenes outside autonomous driving, mainly including:

(1) The existing technology has the problem of insufficient feature extraction in the automatic driving scene with complex structure, resulting in insufficient reconstruction accuracy. The outdoor scene of automatic driving has the characteristics of wide reconstruction range, complex scene structure, and large light changes. In the scene, low-texture and texture-rich areas coexist. The existing technology uses a fixed convolution kernel size to extract features, ignoring the richness of complex texture areas. Scene features; In addition, the existing technology gives equal weight to the feature channel, which fails to filter out the noise data well, resulting in insufficient model reconstruction accuracy.

(2) The existing technology occupies too much video memory space in large-scale outdoor scene reconstruction of autonomous driving. In large-scale outdoor scenes for autonomous driving, the distance between different objects in the scene and the camera varies greatly, so the assumed depth range of the depth map must be set larger; in addition, in order to obtain a better reconstruction effect, the image resolution will also increase. If it is set larger, this will directly lead to the larger size of the constructed cost body, and then the network model will occupy a large amount of video memory space during inference. For example, in the existing MVSNet method, when the image width is set to 1200 pixels, the height is 800 pixels, and the assumed depth range is set to 1024, the algorithm inference stage will occupy about 29Gb of video memory space, which is difficult to run on ordinary consumer-grade graphics cards.

(3) The quality of the input multi-view image directly affects the effect of reconstructing the scene. Most of the data sets used in the existing technology are collected around a certain object or a certain building. This collection method is not suitable for large-scale outdoor scenes of autonomous driving. . For this reason, it is necessary to propose an outdoor image data acquisition and data processing scheme for autonomous driving scenarios.

Contents of the invention

In order to solve the above-mentioned problems existing in the prior art, the present invention provides a method and system for constructing an autonomous driving digital twin scene based on multi-view 3D reconstruction. The feature extraction module in the multi-view 3D reconstruction method of the automatic driving scene proposed by the present invention further extracts the features of the complex texture area by changing the size of the convolution kernel, and assigns different weights to different feature channels. In addition, the extracted feature maps form multiple feature volumes through homography transformation and aggregate them into a cost volume. This method slices the cost volume along the depth dimension, and uses the spatio-temporal recurrent neural network to regularize the slice sequence. While reducing the size of the memory space occupied by the model inference stage in the large-scale scene outside the autopilot, the association between the slices is preserved. In addition, the present invention also provides a UAV-based automatic driving outdoor scene image acquisition scheme, which can effectively and comprehensively acquire the image data required by the multi-view three-dimensional reconstruction method. Finally, the present invention also proposes a self-driving digital twin scene construction system based on multi-view 3D reconstruction, including data collection and processing, camera pose estimation, multi-view 3D reconstruction, point cloud model scale correction, and empty point cloud model fusion 1. Six modules for quantitative evaluation of model accuracy.

The purpose of the present invention is achieved by the following technical solutions: a multi-view 3D reconstruction based automatic driving digital twin scene construction system, the automatic driving digital twin scene construction system includes:

The data collection and processing module (M100), used for collecting and data preprocessing the multi-view images of the autonomous driving scene, and dividing the processed image data into several groups;

The camera pose estimation module (M200) is used to take the collected multi-view images as input, and output the corresponding position and attitude of the camera that captures each image, so as to obtain the sequence of internal and external parameters of the camera;

The multi-view 3D reconstruction module (M300) is used to construct the network model, and extract the feature map sequence of the multi-view image through the network model, and combine the internal and external parameter sequence of the camera to construct the cost body, slice the cost body along the depth dimension, and then The sliced cost volume is processed to obtain the probability volume, and the depth map of the multi-view image is estimated according to the probability volume, and finally the depth map is fused to obtain the three-dimensional dense point cloud of the scene;

The point cloud model scale correction module (M400) is used to use the three feature points processed in the module (M100) and the side lengths of the triangles as input parameters to construct a medium-proportion triangular patch in the virtual three-dimensional space, and in multiple In the 3D dense point cloud of the scene obtained by the view 3D reconstruction module (M300), find the positions of three corresponding feature points, and simultaneously register the three feature points in the virtual point cloud model with the corresponding three points in the triangle patch, Scale transformation of 3D dense point cloud;

The air point cloud model fusion module (M500) is used to divide the three-dimensional dense point cloud obtained by reconstructing the image collected by the drone in the data acquisition and processing module (M100) into an air point cloud model, and reconstruct the other groups of images The three-dimensional dense point cloud is divided into ground point cloud models; this module registers several ground point cloud models to the air point cloud model to form the final autonomous driving digital twin scene model;

The model accuracy quantitative evaluation module (M600) is used to quantitatively evaluate the accuracy of the three-dimensional model of the automatic driving scene, and judge that the accuracy of the three-dimensional model of the automatic driving scene meets the requirements of subsequent automatic driving tasks.

As a preferred solution of the present invention, the method for collecting and processing data by the data collection and processing module (M100) includes the following steps:

S201: Delineate the scope of the automatic driving scene to be reconstructed;

S202: Preset the data collection route, use the unmanned aerial vehicle to fly according to the preset S-shaped route at a fixed flight height, and take scene images at the shooting point;

S203: Lower the flying height of the drone, and shoot around the buildings in the scene in a figure-of-eight manner;

S204: For the buildings next to the road and the section where the road is completely blocked by trees, use a hand-held shooting device to collect data in a way of surrounding shooting;

S205: Preprocessing all the collected image data: adjusting the size of the image to a width of 3000 pixels and a height of 2250 pixels by retaining the most central area of the image, and then downsampling the image to a width of 1600 pixels and a height of 1200 pixels;

S206: divide the preprocessed image data into several groups, wherein the images collected by step S202 and step S203 are divided into one group as the first group of images; Images are grouped individually;

S207: In the real scene covered by each group of images, select the three most obvious feature points, and record the positions of the feature points and the millimeter-level precision side lengths of the formed triangles.

As a preferred solution of the present invention, the camera pose estimation module (M200) includes a retrieval matching unit and an incremental reconstruction unit; the retrieval matching unit takes multi-view images as input, and looks for geometrically verified images with overlapping regions. Image pairs, and calculate the projection of the same point in space on the two images in these image pairs; the incremental reconstruction unit is used to output the corresponding position and attitude of the camera that captures each image.

More preferably, the specific process of the incremental reconstruction unit outputting the corresponding position and attitude of the camera that captures each image is as follows: first, select and register a pair of initial image pairs at the dense position of the multi-view image; The image with the largest number of registration points between the registered images; register the newly added view with the image set whose pose has been determined, and use the PnP problem solving algorithm to estimate the pose of the camera that took the image; then for the newly added view that has been determined Register the unreconstructed spatial points covered by the image, triangulate the image and add new spatial points to the reconstructed spatial point set; finally, perform a process on all currently estimated 3D spatial points and camera poses Bundle adjustment optimization adjustment.

The present invention also provides a scene construction method of the above-mentioned automatic driving digital twin scene construction system, comprising the following steps:

S501: Use the data collection and processing module (M100) to perceive the autonomous driving scene in all directions, and process the collected multi-view image data;

S502: Input the collected multi-view image data into the camera pose estimation module (M200), estimate the corresponding position and pose of the camera that captures each image through retrieval matching and incremental reconstruction methods, and obtain the sequence of internal and external parameters of the camera

和数据采集与处理模块(M100)所采集的图像数据输入到多视图三维重建模块(M300)构建的网络模型中；利用网络模型对图像数据提取图像序列/>

的特征图序列/>

并根据相机内外参数序列/>

和特征图序列

构建特征体序列并聚合成一个代价体；将代价体沿深度维度方向进行切片，并通过网络模型同时处理每个切片及其前后相邻的两个切片，得到描述每一像素在不同深度上概率分布的概率体；S503: Sequence the internal and external parameters of the camera

and the image data collected by the data acquisition and processing module (M100) are input into the network model constructed by the multi-view 3D reconstruction module (M300); the image sequence is extracted from the image data by using the network model/>

The sequence of feature maps />

And according to the sequence of internal and external parameters of the camera />

and a sequence of feature maps

Construct a feature body sequence and aggregate it into a cost body; slice the cost body along the depth dimension, and process each slice and its two adjacent slices through the network model at the same time, and obtain the probability of describing each pixel at different depths the probability body of the distribution;

S504: Adjust the effective depth value in the real depth map to a real value body by means of one-hot encoding, and the real value body is used as a label for supervised learning; input the probability body and the real value body into the original network model, and pass multiple rounds of training way, so that the cross-entropy loss function value between the probability body and the real value body is the smallest, and the trained network model is obtained;

S505: For the input multi-view image sequence, each image is processed by the trained network model to obtain a probability body, and the probability body is adjusted to a depth map; then, the depth map sequence is filtered and fused to obtain the reconstructed 3D dense point cloud of the scene;

S506: Use the point cloud model scale correction module (M400) to construct a medium-proportion triangular patch in the virtual 3D space, register the three feature points in the reconstructed 3D dense point cloud with the corresponding points in the triangular patch, and perform a three-dimensional scene Scale transformation of dense point clouds;

S507: Divide the three-dimensional dense point cloud into an air point cloud model and a ground point cloud model through the air point cloud model fusion module (M500); register several ground point cloud models to the air point cloud model to form the final automatic driving figure twin scene model; and quantitatively evaluate the accuracy of the reconstructed 3D model of the autonomous driving scene to ensure that its accuracy meets the requirements of subsequent autonomous driving tasks.

的获取具体为：通过网络模型学习到卷积核方向向量的偏移量，使得卷积核能适应不同纹理结构的区域，提取更为细致的特征；其次，将不同尺寸的特征图上采样到原输入图像大小，并将它们连接到一起组成一张具有32通道的特征图；然后，将每个特征通道的二维信息u_c(i,j)压缩成一维实数z_c，并进行两级全连接；最后，使用sigmoid函数将每个实数z_c的范围限制在[0,1]范围内，使得特征图的每个通道具有不同的权重，削弱匹配过程中的噪声数据与无关特征；对每张输入图像均重复上述操作步骤，得到特征图序列/>

As a preferred solution of the present invention, the sequence of feature maps in step S503

The acquisition is as follows: the offset of the convolution kernel direction vector is learned through the network model, so that the convolution kernel can adapt to regions with different texture structures and extract more detailed features; secondly, the feature maps of different sizes are sampled to the original Input the size of the image, and connect them together to form a feature map with 32 channels; then, compress the two-dimensional information u _c (i,j) of each feature channel into a one-dimensional real number z _c , and perform two-stage full Connection; finally, use the sigmoid function to limit the range of each real number z _c to the range [0,1], so that each channel of the feature map has different weights, weakening the noise data and irrelevant features in the matching process; for each Repeat the above operation steps for each input image to obtain a sequence of feature maps />

然后根据相机内外参数序列

将所有的特征图通过单应变换投影到参考图像下的若干个平行平面上，构成N-1个特征体/>

最后，通过聚合函数将特征体聚合成一个代价体。As a preferred solution of the present invention, the aggregation of the cost body in step S503 is specifically: select an image in the feature map sequence as the reference feature map F ₁ , and the remaining feature maps as the source feature map

Then according to the sequence of internal and external parameters of the camera

Project all feature maps to several parallel planes under the reference image through homography transformation to form N-1 feature bodies/>

Finally, the feature volume is aggregated into a cost volume through the aggregation function.

As a preferred solution of the present invention, the formation of the probability body in step S503 is specifically: cutting the cost body into D slices, where D is the depth prior, indicating that the depth value can take any value from 0 to D; The volume slice sequence is regarded as a time series, which is sent to the spatiotemporal recurrent neural network in the network model for regularization. The spatiotemporal recurrent neural network uses ST-LSTM to transfer the storage state in time series (horizontal direction) and space domain (vertical direction), and preserve the slice sequence The connection between them reduces the occurrence of multiple peaks in the probability body; in the horizontal direction, the first layer of units at a certain moment accepts the hidden state and memory state of the last layer of units at the previous moment, and transmits them layer by layer in the vertical direction; Finally, the softmax normalization operation is used to output the probability value of each pixel at the depth d∈[0,D] to form a probability volume.

As a preferred solution of the present invention, step S505 is specifically: perform an argmax operation on the probability body obtained by inference of the trained network model for each image to obtain a sequence of depth maps, and filter out the For depth maps with low confidence, the depth map is finally fused into a three-dimensional dense point cloud through the formula P=dM ^-1 K ^-1 p; where p is the pixel coordinate, d is the depth value obtained by network model reasoning, and P is the world coordinate system The three-dimensional coordinates below.

Compared with the prior art, the present invention has the following beneficial technical effects:

(1) The present invention proposes a multi-view 3D reconstruction method for large-scale scenes outside the automatic driving. This method learns the offset of the convolution kernel direction vector through the sub-network model, and then has different convolution kernel sizes when processing different image parts, so that it has different receptive fields when facing regions with different texture complexity. It is more suitable for areas with complex textures. By assigning different weights to different channels in the extracted feature map, this method enhances important features in the matching process, weakens noise data and irrelevant features, and improves the accuracy and robustness of feature extraction. Aiming at the characteristics of complex scene structure, low-texture and texture-rich areas in the outdoor scene of automatic driving, the present invention proposes a new feature extraction module, which overcomes the lack of feature extraction in complex texture areas in the prior art and is greatly affected by noise data and other issues, thereby improving the reconstruction accuracy of the autonomous driving scene model.

(2) The multi-view 3D reconstruction method of the autonomous driving scene proposed by the present invention slices the cost body along the depth dimension, and then loads the slice sequence into the spatio-temporal recurrent neural network, so that the size of the memory occupied by the inference stage is the same as the assumed depth range d irrelevant, and then the network model can be used for reconstruction of large-scale scenes outside the autonomous driving assuming a wide range of depths. The present invention uses ST-LSTM to transfer the storage state in the horizontal and vertical directions, which reduces the space occupied by the video memory while retaining the connection between the slice sequences, and reduces the occurrence of multiple peaks in the probability body in the prior art, and improves the depth prediction accuracy.

(3) The present invention proposes an image data collection and data processing scheme for automatic driving scenes. The scheme presets the UAV flight route and shooting points by calculating the image overlap rate, and combines S-shaped and figure-of-eight flying around The way of flight shooting ensures the quality of the collected multi-view images. In addition, in order to solve the problem that some lanes are blocked by trees in the aerial view, the solution also uses a handheld camera device to collect image data of the scene near the ground, and groups the captured images to provide an effective and comprehensive solution for the outdoor image data of the autonomous driving scene. Acquisition and data processing provide new solutions.

(4) The present invention proposes a self-driving digital twin scene construction system based on multi-view 3D reconstruction, which provides the whole process of autonomous driving digital twin scene reconstruction and accuracy evaluation, including data collection and processing, camera pose estimation , Multi-view 3D reconstruction, point cloud model scale correction, empty point cloud model fusion, model accuracy quantitative evaluation module. The system can efficiently and comprehensively collect image data of the scene outside the autonomous driving, and estimate the position and pose of the camera that captured these images. In addition, the system can reconstruct the 3D model of the autonomous driving scene with a smaller video memory footprint and stronger feature extraction capabilities in complex texture areas, and improves the integrity of the scene model through the integration of cloud models of empty locations. It also provides A quantitative evaluation method evaluates the accuracy of the reconstructed scene model.

Description of drawings

Fig. 1 is the deep neural network model structure of the multi-view 3D reconstruction method for the large-scale scene outside the automatic driving proposed by the present invention.

Figure 2 is the feature extraction module in Figure 1.

Fig. 3 is a flow chart of the method for collecting scene data of automatic driving proposed by the present invention.

Fig. 4 is a schematic diagram of the method for collecting data of an automatic driving scene proposed by the present invention; S202, S203, and S204 in the figure are the corresponding operation steps in the embodiment.

Fig. 5 is a schematic diagram of image data processing and grouping in steps S205 and S206 in the embodiment.

Fig. 6 is a module structure diagram of the autonomous driving digital twin scene construction system proposed by the present invention.

FIG. 7 is a schematic diagram of triangulation adding new spatial points in the camera pose estimation module M200 in the system.

Fig. 8 is a schematic diagram of the spatial triangular patch constructed in the scale correction module M400 of the point cloud model in the system.

Fig. 9 is a model fusion effect diagram of the M500 model fusion module of the hollow location cloud model in the system.

Figure 10 is a diagram of the accuracy quantitative evaluation results of the system's hollow location cloud model fusion module M600.

Fig. 11 is a scene effect diagram reconstructed by the autonomous driving digital twin scene construction system proposed by the present invention.

Fig. 12 is an effect diagram of a scene reconstructed by the autonomous driving digital twin scene construction system proposed by the present invention.

Detailed ways

The present invention will be further elaborated and described below in combination with specific embodiments. The embodiments are merely exemplary of the disclosure and do not delineate the scope of limitation. The technical features of the various implementations in the present invention can be combined accordingly on the premise that there is no conflict with each other.

The structure of the self-driving digital twin scene construction system based on multi-view 3D reconstruction proposed by the present invention is shown in Figure 6, which consists of 6 modules. In this embodiment, the self-driving digital twin scene is constructed according to these 6 modules:

The data collection and processing module (M100), used for collecting and data preprocessing the multi-view images of the autonomous driving scene, and dividing the processed image data into several groups;

Camera pose estimation module M200: In this embodiment, the images obtained by the M100 module are input into this module, and the camera poses of these images are obtained through the processing of the retrieval matching unit M201 and the incremental reconstruction unit M202. Specifically, in this embodiment, the image is input into the retrieval and matching unit, first, the feature point detection is performed on the image of each viewing angle and the SIFT feature descriptor of the feature point is extracted, and then a group of possible matching image pairs C={(I _a ,I _b )|I _a ,I _b ∈I,a<b}, calculate the relationship matrix M _ab ∈F _a ×F _b of the corresponding features of the two images. Finally, in this embodiment, the eight-point method is used to estimate the fundamental matrix of the camera, and the RANSAC method is used to remove the corresponding relationship determined to be outliers when estimating the fundamental matrix, so as to construct a scene graph. In this embodiment, the scene graph is input to the incremental reconstruction unit. In this embodiment, the image with the largest number of registration points between the currently registered images is selected, and the newly added view is registered with the image set whose pose has been determined. allow. Optionally, this embodiment uses the DLT algorithm to solve the PnP problem, obtains 2n constraint equations according to the world coordinates of n points and the pixel coordinates in the image, and then uses SVD to solve the overdetermined equations to realize the estimation of the camera pose matrix. As shown in Figure 7, in this embodiment, according to the projection of the same point in space in multiple images with different viewing angles, the unreconstructed spatial point covered by the newly added registered image is found, and it is added to the reconstructed out of the set of spatial points. Finally, in this embodiment, an optimal adjustment of bundle adjustment is performed on all currently estimated three-dimensional space points and camera poses. Specifically, this embodiment minimizes the reprojection error in Formula 1 by adjusting the camera pose and estimated scene point positions.

Wherein P _i is the i-th three-dimensional point in space, m represents the number of three-dimensional points in space, n represents the number of cameras, R and t are the rotation matrix and translation matrix in n cameras, e _ij =π(P _i ,R _j ,t _j )-p _ij represents the projection error of the i-th 3D point in the space at the j-th camera.

Multi-view 3D reconstruction module M300: used to build a network model, and extract the feature map sequence of multi-view images through the network model, and combine the internal and external parameter sequences of the camera to construct the cost body, slice the cost body along the depth dimension, and then slice the The cost body is processed to obtain the probability volume, and the depth map of the multi-view image is estimated according to the probability volume, and finally the depth map is fused to obtain the three-dimensional dense point cloud of the scene;

Point cloud model scale correction module M400: as shown in Figure 8, it is used to use the three feature points processed in the module (M100) and the side lengths of the triangles as input parameters to construct a triangular patch of medium proportion in the virtual three-dimensional space , and in the 3D dense point cloud of the scene obtained by the multi-view 3D reconstruction module (M300), find the positions of three corresponding feature points, and compare the three feature points in the virtual point cloud model with the corresponding three points in the triangle patch Simultaneously register and perform scale transformation on the three-dimensional dense point cloud; in this embodiment, the three feature points obtained in the module M100 and the side lengths of the triangles are used as input parameters to construct an equal-proportion triangle patch R in the virtual three-dimensional space ₀ R ₁ T ₂ . Then, in this embodiment, in the reconstructed 3D point cloud model of the scene, the positions of three corresponding feature points are found, and the three feature points in the virtual point cloud model are simultaneously registered with the corresponding three points in the triangle patch, Perform scale transformation on the reconstructed 3D point cloud model. Specifically, in this embodiment, the scale of the reconstructed 3D point cloud model is enlarged by 1.1 times.

Air point cloud model fusion module M500: used to divide the 3D dense point cloud obtained by reconstructing the image collected by the drone in the data acquisition and processing module (M100) into an air point cloud model, and reconstruct the 3D dense point cloud obtained by other groups of images The point cloud is divided into ground point cloud models; this module registers several ground point cloud models to the air point cloud model to form the final autonomous driving digital twin scene model; this embodiment finds Three feature points that also exist in the aerial point cloud model, these feature points are distributed on the edge of the building and the intersection of the three facades. According to these matching feature points, this embodiment respectively performs translation and rotation matrix transformation on the two ground point cloud models, and fuses them into the air point cloud. The fusion result is shown in Figure 9, where the darker parts (the buildings in the picture and the road covered by trees below the picture) are the ground point cloud model before fusion.

Model Accuracy Quantitative Evaluation Module M600: Used to quantitatively evaluate the accuracy of the three-dimensional model of the automatic driving scene, and judge that the accuracy of the three-dimensional model of the automatic driving scene meets the requirements of subsequent automatic driving tasks; The positions and distances of the 40 feature point pairs related to the lane, find the corresponding feature point pairs in the reconstructed scene 3D point cloud model, measure the distance of the corresponding feature point pairs, and compare each of the virtual 3D point cloud models. The absolute error value and error percentage between the distance of each point pair and the distance in the actual scene. The evaluation results of this embodiment are shown in Figure 10. Under the 40 feature point pairs of the scene, the maximum absolute error in the 3D point cloud model of the autonomous driving digital twin scene constructed in this embodiment is 6.08 cm, and the average absolute error is 2.5 cm. cm, the maximum percentage is 2.361%, and the average percentage is 0.549%, where the percentage measure refers to the ratio of the error to the point-to-point distance in the actual scene. The autonomous driving digital twin scene constructed in this embodiment is shown in Fig. 11 and Fig. 12 .

According to the multi-view three-dimensional reconstruction method for large-scale scenes outside the automatic driving provided by the first aspect of the present invention, this embodiment includes the following steps:

S1: Use the data acquisition and processing module (M100) to perceive the autonomous driving scene in all directions, and process the collected multi-view image data;

S2: Input the collected multi-view image data into the camera pose estimation module (M200), estimate the corresponding position and pose of the camera that took each image through retrieval matching and incremental reconstruction methods, and obtain the sequence of internal and external parameters of the camera

和数据采集与处理模块(M100)所采集的图像数据输入到多视图三维重建模块(M300)构建的网络模型中；利用网络模型对图像数据提取图像序列/>

的特征图序列/>

并根据相机内外参数序列/>

和特征图序列

构建特征体序列并聚合成一个代价体；将代价体沿深度维度方向进行切片，并通过网络模型同时处理每个切片及其前后相邻的两个切片，得到描述每一像素在不同深度上概率分布的概率体；S3: Sequence the internal and external parameters of the camera

and the image data collected by the data acquisition and processing module (M100) are input into the network model constructed by the multi-view 3D reconstruction module (M300); the image sequence is extracted from the image data by using the network model/>

The sequence of feature maps />

And according to the sequence of internal and external parameters of the camera />

and a sequence of feature maps

Construct a feature body sequence and aggregate it into a cost body; slice the cost body along the depth dimension, and process each slice and its two adjacent slices through the network model at the same time, and obtain the probability of describing each pixel at different depths the probability body of the distribution;

S4: The effective depth value in the real depth map is adjusted to the real value body by one-hot encoding, and the real value body is used as a label for supervised learning; the probability body and the real value body are input into the original network model, and through multiple rounds of training way, so that the cross-entropy loss function value between the probability body and the real value body is the smallest, and the trained network model is obtained;

S5: For the input multi-view image sequence, each image is processed by the trained network model to obtain the probability body, and the probability body is adjusted into a depth map; then, the depth map sequence is filtered and fused to obtain the reconstructed 3D dense point cloud of the scene;

S6: Use the point cloud model scale correction module (M400) to construct a medium-proportion triangular patch in the virtual 3D space, register the three feature points in the reconstructed 3D dense point cloud with the corresponding points in the triangular patch, and perform three-dimensional analysis of the scene Scale transformation of dense point clouds;

S7: Divide the 3D dense point cloud into an air point cloud model and a ground point cloud model through the space point cloud model fusion module (M500); register several ground point cloud models to the air point cloud model to form the final autopilot digital twin scene model; and quantitatively evaluate the accuracy of the reconstructed 3D model of the autonomous driving scene to ensure that its accuracy meets the requirements of subsequent autonomous driving tasks.

的特征图。由于传统卷积操作卷积核大小固定，感受野相同，在面对纹理复杂区域时，往往忽视了很多重要特征。因此，本实施例通过子网络模型学习到卷积核方向向量的偏移量，进一步提取纹理复杂区域的特征。本实施例公式2将提取到的特征图X处理成由不同通道权重组成的特征图X′。具体地，本实施例通过插值的方式将不同尺寸的特征图上采样到W×H的分辨率，并连接成具有32个通道的特征图；然后根据公式3将每个通道的二维特征u_c(i,j)压缩成一维实数z_c；然后根据公式4对32个实数z_c进行两级的全连接W₁,W₂，最后用sigmoid函数将每个实数的范围限制在[0,1]范围内，进而增强匹配过程中的重要特征，削弱无关特征，提高特征提取的准确性。Step 3 in this embodiment is specifically as follows: the size of the input image is W=1600 pixels in width and H=1200 pixels in height. As shown in Figure 2, this embodiment performs three traditional convolution operations on each input image to obtain three sizes of

feature map of . Due to the fixed size of the convolution kernel and the same receptive field in the traditional convolution operation, many important features are often ignored when facing areas with complex textures. Therefore, in this embodiment, the subnetwork model is used to learn the offset of the direction vector of the convolution kernel to further extract the features of the region with complex textures. Formula 2 of this embodiment processes the extracted feature map X into a feature map X′ composed of different channel weights. Specifically, this embodiment upsamples feature maps of different sizes to a resolution of W×H through interpolation, and connects them into a feature map with 32 channels; then according to formula 3, the two-dimensional feature u of each channel _c (i, j) is compressed into a one-dimensional real number z _c ; then according to formula 4, two-stage fully connected W ₁ , W ₂ is performed on 32 real numbers z _c , and finally the range of each real number is limited to [0, 1], thereby enhancing the important features in the matching process, weakening irrelevant features, and improving the accuracy of feature extraction.

X′＝F _scale {[F _sq [f(upsaaple(X _C ))],s _c } (2)

s _c =F _ex (z,W)=σ(g(z,W))=σ(W ₂ δ(W ₁ z)) (4)

和特征图序列/>

构建代价体。本实施例选择输入的其中一张特征图作为参考图像特征图，将其它特征图视为源图像特征图，并假设深度范围值为d。因此，本实施例通过可微的单应变换将N-1张源图像特征图映射到参考图像特征图下的d个平行平面上，以得到N-1个特征体/>

这些特征体刻画了在深度值为d的平面上第i个特征图F_i与参考特征图F₁之间的单应变换关系。可微的单应变换具体内容可参考Yao Y,et al.MVSNET:depth inference for unstructured multi-viewstereo[C],Springer,European Conference on Computer Vision(ECCV),Munich,2018:785-801.对于这些特征体，本实施例按照公式5进行处理，聚合成代价体C，其中/>

为代价体序列的平均值。Then through the input camera internal and external parameter sequence

and sequence of feature maps />

Build cost body. In this embodiment, one of the input feature maps is selected as the reference image feature map, the other feature maps are regarded as source image feature maps, and the depth range value is assumed to be d. Therefore, in this embodiment, N-1 source image feature maps are mapped to d parallel planes under the reference image feature map through differentiable homography transformation to obtain N-1 feature bodies/>

These feature volumes describe the homography transformation relationship between the i-th feature map F _i and the reference feature map F ₁ on a plane with a depth value of d. The specific content of differentiable homography transformation can refer to Yao Y, et al. MVSNET: depth inference for unstructured multi-viewstereo [C], Springer, European Conference on Computer Vision (ECCV), Munich, 2018: 785-801. For these The feature body, in this embodiment, is processed according to formula 5, and aggregated into a cost body C, where />

is the average value of the cost body sequence.

Cut the obtained cost body with a volume size of V=W·H·D·F into D pieces of size W·H·F, where W and H are the size of the feature map, F is the number of channels of the feature map, D is the depth prior. The sliced cost body is not an isolated individual. In this embodiment, the slice sequence of the cost body is regarded as a time series, which is sent to the spatio-temporal recurrent neural network for regularization. The spatio-temporal recurrent neural network used in this implementation is composed of several ST-LSTM memory units. It is composed of three moments in the horizontal direction, representing the cost body slices input at the previous moment, the current moment, and the next moment respectively. It consists of four layers in the direction, and each layer is connected by an ST-LSTM memory unit, and the first layer unit at the current moment accepts the hidden state and memory state of the last layer unit at the previous moment. The specific structure of the ST-LSTM unit can refer to Y.Wang et al.,"PredRNN:A Recurrent Neural Network for Spatiotemporal Predictive Learning,"inIEEE Transactions on Pattern Analysis and Machine Intelligence, doi:10.1109/TPAMI.2022.3165153. Therefore, this implementation In this example, the connection of the slice sequence of the cost volume is established through the above method, which reduces the occurrence of multi-peaks in the obtained probability volume and improves the accuracy of depth prediction. In addition, the network model only processes three slices at a time, which reduces the memory space occupied during model inference. Specifically, this embodiment uses the Pytorch deep learning framework to perform model inference on a graphics card modeled as NVIDIA GeForce GTX 3060 (6GB of video memory), and implement the above process.

The effective depth value in the real depth map is adjusted to the real value body by means of one-hot encoding. The optimization objective function adopted in the model training process of this embodiment is formula 6, wherein G ^(d) (p) and P ^{(d )} (p) are the depth real value and estimated value at the depth assumption d and pixel p respectively; since the depth value in the real depth map is incomplete, in this embodiment, only the effective depth in the depth map is used Valued pixels are supervised.

Filter and fuse the depth map. In this embodiment, the depth map is filtered according to the photometric consistency criterion and the geometric consistency criterion. Specifically, in this embodiment, points whose probability value corresponding to the estimated depth value is lower than 0.8 are regarded as outliers, and valid pixels are required to satisfy the inequality described in formula 7

, where p _i is the projection of the depth value d ₁ at the pixel point p ₁ of the reference image at the point p _i of the neighborhood view, and the corresponding depth value is d _i , and p _reproj is the depth value d i at the point _{p i} in the reference image Reprojection on , the corresponding depth value is d _reproj .

As shown in FIG. 3 , according to the automatic driving scene data collection and data processing method provided by the second aspect of the present invention, this embodiment includes the following steps:

S201: Define the scope of the autonomous driving scene to be reconstructed.

S202: Presetting the data collection route, using the unmanned aerial vehicle to fly according to the preset route at a fixed flight height, and shooting scene images at the shooting point.

S203: Lower the flying height of the drone, and shoot around the buildings in the scene in a figure-of-eight manner.

S204: For the buildings next to the road and the road sections where the road is completely covered by trees, use a hand-held shooting device to collect data in a surround shooting manner.

S205: Preprocessing all the collected image data.

S206: Divide the image data collected and processed according to the above steps into several groups.

S207: Select feature points in the real scene covered by each group of images.

In this embodiment, according to step S201, an area of about 40,000 square meters that allows consumer-grade drones to fly and does not have a large amount of strong reflective materials is selected, and the noon time period with sufficient lighting conditions, less traffic, and less traffic is selected to use the drone. The man-machine takes pictures.

As shown in Figure 4, according to step S202, this embodiment fixes the flying height of the UAV to 90 meters, the 35mm equivalent focal length of the airborne camera used is 24mm, the frame size is 24×36mm, and the heading overlap rate and side The overlap rate in all directions is set to 85%, and the shooting point position is calculated according to formulas 8 and 9.

Among them, fw _overlap is the heading overlap rate, side _overlap is the side overlap rate, frame _w and frame _l are the width and height of the frame size, and focal is the equivalent focal length. In this embodiment, it is calculated that along the flight direction of the drone, a shooting point is set every 13.5 meters, and the lateral spacing of the S-shaped flight route is 20 meters. At each shooting point, this embodiment shoots one image in each of the five directions of forward oblique front, forward oblique back, forward oblique left, forward oblique right, and directly below, and a total of 380 valid images are collected.

According to step S203, in this embodiment, the flying height of the drone is adjusted to 35 meters, around the buildings in the scene, and the obvious buildings in the scene are photographed in the way of flying around the scene as shown in Figure 4, as shown in Figure 4 Each point in the figure-of-eight flight route is a shooting point, and the number of valid images collected is 307.

According to step S204, this embodiment uses a shooting device with a single camera to collect data on buildings next to the road and road sections where the road is completely blocked by trees as shown in Figure 4, and a total of 181 effective images are collected.

According to step S205, this embodiment performs batch processing on the image data, as shown on the left side of Figure 5, retains the most central area of the original image, adjusts the image size to a width of 3000 pixels, and a height of 2250 pixels, and then downsamples the image to 1600 pixels wide and 1200 pixels high. According to step S206, this embodiment divides the processed image data into three groups, as shown on the right side of Figure 5, the images collected by the drone are one group, the roads completely blocked by trees are one group, and the buildings are one group. as a group. According to step S207, after the above steps are completed, in order to meet the requirements of the automatic driving task, this embodiment selects three cuboid stone pier vertices and lane line corner points in the scene as feature points for the point cloud scale correction module in the system, and In the scene, 40 pairs of feature points are uniformly selected for quantitative evaluation of the point cloud model accuracy in the system.

The above-mentioned embodiments only express several implementation modes of the present invention, and the description thereof is relatively specific and detailed, but should not be construed as limiting the patent scope of the present invention. For those skilled in the art, without departing from the concept of the present invention, several modifications and improvements can be made, and these all belong to the protection scope of the present invention.

Claims (9)

1. An autopilot digital twin scene construction system based on multi-view three-dimensional reconstruction, characterized in that the autopilot digital twin scene construction system comprises:

the data acquisition and processing module (M100) is used for acquiring and preprocessing the multi-view images of the automatic driving scene and dividing the processed image data into a plurality of groups;

a camera pose estimation module (M200) for taking the collected multi-view images as input, and outputting the corresponding position and pose of a camera for shooting each image, thereby obtaining the internal and external parameter sequences of the camera;

the multi-view three-dimensional reconstruction module (M300) is used for constructing a network model, extracting a characteristic image sequence of a multi-view image through the network model, constructing a cost body by combining an internal and external parameter sequence of a camera, slicing the cost body along the depth dimension direction, processing the sliced cost body to obtain a probability body, estimating a depth image of the multi-view image according to the probability body, and finally fusing the depth image to obtain a three-dimensional dense point cloud of the scene;

The point cloud model scale correction module (M400) is used for constructing a triangle patch with a medium proportion in a virtual three-dimensional space by taking three characteristic points obtained by processing in the module (M100) and side lengths of triangles formed by the three characteristic points as input parameters, finding out positions of three corresponding characteristic points in a scene three-dimensional dense point cloud obtained by the multi-view three-dimensional reconstruction module (M300), registering the three characteristic points in the virtual point cloud model with the corresponding three points in the triangle patch at the same time, and performing scale transformation on the three-dimensional dense point cloud;

the space-point cloud model fusion module (M500) is used for dividing the three-dimensional dense point cloud obtained by reconstructing the image acquired by the unmanned aerial vehicle in the data acquisition and processing module (M100) into a space-point cloud model and dividing the three-dimensional dense point cloud obtained by reconstructing other groups of images into a ground point cloud model; registering a plurality of ground point cloud models to an aerial point cloud model by the module to form a final automatic driving digital twin scene model;

and the model precision quantitative evaluation module (M600) is used for quantitatively evaluating the precision of the three-dimensional model of the automatic driving scene and judging that the precision of the three-dimensional model of the automatic driving scene meets the requirement of a subsequent automatic driving task.

2. The automated driving digital twinning scenario construction system according to claim 1, wherein the method of data acquisition and processing by the data acquisition and processing module (M100) comprises the steps of:

s201, demarcating an automatic driving scene range to be rebuilt;

s202, presetting a data acquisition route, flying according to a preset S-shaped route at a fixed flying height by using an unmanned aerial vehicle, and shooting scene images at shooting points;

s203, lowering the flight height of the unmanned aerial vehicle, and shooting around a building in a scene in a splayed flight-around mode;

s204, for buildings beside the road and road sections where the road is completely covered by trees, acquiring data in a surrounding shooting mode by using a handheld shooting device;

s205, preprocessing all collected image data: adjusting the image size to 3000 pixels wide and 2250 pixels high by reserving the most central area of the image, and then downsampling the image to 1600 pixels wide and 1200 pixels high;

s206, dividing the preprocessed image data into a plurality of groups, wherein the images acquired in the step S202 and the step S203 are divided into a group to be used as a first group of images; grouping the images photographed for each building or road section in step S204 individually;

S207, selecting three most obvious characteristic points in the real scene covered by each group of images, and recording the positions of the characteristic points and the millimeter-level precision side length of the formed triangle.

3. The automated driving digital twin scene building system of claim 1, wherein the camera pose estimation module (M200) comprises a search matching unit and an incremental reconstruction unit; the searching and matching unit takes multi-view images as input, searches for image pairs which are geometrically verified and have overlapping areas, and calculates the projection of the same point in space on two images in the image pairs; the incremental reconstruction unit is used for outputting the corresponding position and posture of the camera for shooting each image.

4. The automated driving digital twin scene building system according to claim 3, wherein the specific process of outputting the position and posture corresponding to the camera capturing each image by the incremental reconstruction unit is: first selecting and registering a pair of initial image pairs at a multi-view image dense location; then selecting the image with the largest registration point number with the currently registered image; registering the newly added view with the image set with the determined pose, and estimating the pose of a camera shooting the image by using a PnP problem solving algorithm; then, for the unreconstructed spatial points covered by the newly added registered image, triangulating the image and adding the new spatial points to the reconstructed spatial point set; and finally, performing one-time beam adjustment optimization adjustment on all the current estimated three-dimensional space points and camera pose.

5. A scene construction method of an autopilot digital twin scene construction system according to claim 1, comprising the steps of:

s501: the data acquisition and processing module (M100) is used for carrying out omnibearing sensing on an automatic driving scene and processing the acquired multi-view image data;

s502: inputting the acquired multi-view image data into a camera pose estimation module (M200), estimating the position and the pose corresponding to a camera shooting each image by searching, matching and incremental reconstruction methods, and obtaining an internal and external parameter sequence of the camera

S503: camera inner and outer parameter sequence

Inputting the image data acquired by the data acquisition and processing module (M100) into a network model constructed by the multi-view three-dimensional reconstruction module (M300); extracting an image sequence from image data using a network model>

Is->

And according to the camera in-out parameter sequence->

And the sequence of feature maps->

Constructing a feature sequence and polymerizing the feature sequence into a cost body; slicing the cost body along the depth dimension direction, and simultaneously processing each slice and two adjacent slices in front and behind through a network model to obtain probability bodies describing probability distribution of each pixel at different depths;

S504, adjusting the effective depth value in the real depth map into a real value body by a single-heat coding mode, wherein the real value body is used as a label for supervised learning; inputting the probability body and the true value body into an original network model, and obtaining a trained network model by a multi-round training mode to minimize the cross entropy loss function value between the probability body and the true value body;

s505, processing each image of the input multi-view image sequence through a trained network model to obtain a probability body, and adjusting the probability body into a depth map; then, filtering and fusing the depth map sequence to obtain a reconstructed three-dimensional dense point cloud of the scene;

s506: constructing a triangle patch with a medium proportion in a virtual three-dimensional space through a point cloud model scale correction module (M400), registering three characteristic points in the reconstructed three-dimensional dense point cloud with corresponding points in the triangle patch, and performing scale transformation on the three-dimensional dense point cloud of the scene;

s507: dividing the three-dimensional dense point cloud into an air point cloud model and a ground point cloud model by an air-to-ground point cloud model fusion module (M500); registering a plurality of ground point cloud models to an aerial point cloud model to form a final automatic driving digital twin scene model; and quantitatively evaluating the precision of the reconstructed three-dimensional model of the automatic driving scene to ensure that the precision meets the requirements of subsequent automatic driving tasks.

6. The scene construction method according to claim 5, wherein the feature map sequence in step S503

The acquisition of (1) is specifically as follows: the offset of the convolution kernel direction vector is learned through the network model, so that the convolution kernel can adapt to areas with different texture structures, and finer features are extracted; secondly, up-sampling the feature images with different sizes to the original input image size, and connecting the feature images to form a feature image with 32 channels; then, the two-dimensional information u of each characteristic channel is calculated _c (i, j) compressing into one-dimensional real number z _c And performing two-stage full connection; finally, each real number z is determined using a sigmoid function _c Is limited to [0,1 ]]Within the range, each channel of the feature map has different weights, weakening the matchingNoise data and irrelevant features in the process; repeating the above steps for each input image to obtain a feature map sequence +.>

7. The scene construction method according to claim 5, wherein the aggregation of the cost volumes in step S503 is specifically: selecting an image of the sequence of feature maps as reference feature map F ₁ The rest feature images are used as source feature images

Then according to the camera inner and outer parameter sequence->

Projecting all feature images onto a plurality of parallel planes under a reference image through homography transformation to form N-1 feature bodies +. >

Finally, the feature bodies are aggregated into a cost body through an aggregation function.

8. The scene construction method according to claim 5, wherein the forming of the probability volume in step S503 is specifically: cutting the cost body into D pieces, wherein D is depth priori, and the depth value can be any one value from 0 to D; then, the cost body slice sequence is regarded as a time sequence, the time-space recurrent neural network in the network model is sent to regularize, the time-space recurrent neural network uses ST-LSTM to transfer the storage state in the time sequence, namely the horizontal direction and the airspace, namely the vertical direction, the relation between slice sequences is reserved, and the situation that the probability body has multiple peaks is reduced; in the horizontal direction, the first layer unit at a certain moment receives the hidden state and the memory state of the last layer unit at the previous moment and transmits the hidden state and the memory state layer by layer in the vertical direction; finally, a softmax normalization operation is used for outputting probability values of each pixel at the depth d E [0, D ] to form a probability body.

9. The scene construction method according to claim 5, wherein step S505 is specifically: performing argmax operation on a probability body obtained by reasoning a trained network model of each image to obtain a depth map sequence, filtering out a depth map with low confidence coefficient based on a luminosity consistency criterion and a geometric consistency criterion, and finally obtaining a depth map sequence through the formula p=dm ^-1 K ^-1 p fusing the depth map into a three-dimensional dense point cloud; where P is the pixel coordinate, d is the depth value inferred by the network model, and P is the three-dimensional coordinate in the world coordinate system.

Images