CN111860493B - Target detection method and device based on point cloud data - Google Patents

Abstract

This application provides a target detection method and device based on point cloud data, and relates to the technical field of target detection. The method includes: obtaining original point cloud data and each initial target detection frame output by the initial target detection network, and obtaining information on each initial target detection frame. Extract point clouds within the preset frame range around each initial target detection frame from the original point cloud data; based on the points in the original point cloud data within each initial target detection frame and the point clouds within the preset frame range outside the initial target detection frame The points in and the information of each initial target detection frame are generated to generate the input data of the neural network; the input data is input into the pre-trained neural network for processing, and the detection frame corresponding to each initial target detection frame is obtained based on the output of the pre-trained neural network. results and target category results. The embodiments of the present application can improve the performance of target detection.

Description

A target detection method and device based on point cloud data

Technical field

The present application relates to the field of target detection technology, and in particular, to a target detection method and device based on point cloud data.

Background technique

Currently, with the development of autonomous driving technology and mobile robot technology, lidar equipment has been widely used in autonomous vehicles and mobile robots. In order to ensure the normal operation of autonomous vehicles and mobile robots, it is generally necessary to collect point cloud data of the surrounding environment through lidar to help autonomous vehicles and mobile robots perceive their surrounding environment. In order to better perceive their surrounding environment, current self-driving vehicles and mobile robots generally need to perform target detection on various objects in their surrounding environment. One way is to use point cloud data collected by lidar.

Currently, existing technologies include inputting point cloud data into a target detection network to perform target detection, such as using Sparsely Embedded Convolutional Detection (SECOND), new point cloud encoders and networks. (PointPillars), Multi-View3D (MV3D for short), Point Voxel-RCNN (PV-RCNN for short) or a new three-dimensional object detection network (Point RCNN). However, the target detection performed by the above-mentioned existing target detection networks has the problem of poor detection performance. In order to improve the performance of target detection, this application will provide a target detection solution based on point cloud data.

Contents of the invention

Embodiments of the present application provide a target detection method and device based on point cloud data to improve the performance of target detection.

In order to achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

A first aspect of the embodiments of this application provides a target detection method based on point cloud data, including:

Obtain the original point cloud data and each initial target detection frame output by the initial target detection network, and obtain the information of each initial target detection frame;

Extract point clouds within the preset frame range around each initial target detection frame from the original point cloud data;

Generate the input data of the neural network based on the points in the original point cloud data within each initial target detection frame, the points in the point cloud within the preset frame outside the initial target detection frame, and the information on each initial target detection frame;

The input data is input into a pre-trained neural network for processing, and the detection frame results and target category results corresponding to each initial target detection frame are obtained according to the output of the pre-trained neural network.

A second aspect of the embodiment of the present application provides a target detection device based on point cloud data, including:

The initial information acquisition unit is used to obtain the original point cloud data and each initial target detection frame output by the initial target detection network, and obtain the information of each initial target detection frame;

A point cloud extraction unit configured to extract point clouds within the preset frame range around each initial target detection frame from the original point cloud data;

The input data generation unit is used to generate the neural network based on the points in the original point cloud data within each initial target detection frame, the points in the point cloud within the preset frame outside the initial target detection frame, and the information on each initial target detection frame. Input data;

The result generation unit is used to input the input data into a pre-trained neural network for processing, and obtain the detection frame results and target category results corresponding to each initial target detection frame based on the output of the pre-trained neural network.

A third aspect of the embodiments of the present application provides a computer-readable storage medium, including a program or instructions. When the program or instructions are run on a computer, the method described in the first aspect is implemented.

A fourth aspect of the embodiments of the present application provides a computer program product containing instructions, which when the computer program product is run on a computer, causes the computer to execute the method described in the first aspect.

A fifth aspect of the embodiment of the present application provides a chip system, including a processor, the processor is coupled to a memory, and the memory stores program instructions. When the program instructions stored in the memory are executed by the processor When implementing the method described in the first aspect above.

A sixth aspect of the embodiments of the present application provides a computer server, including a memory, and one or more processors communicatively connected to the memory;

The memory stores instructions that can be executed by the one or more processors, and the instructions are executed by the one or more processors, so that the one or more processors implement the first aspect described above. method described.

Embodiments of the present application provide a target detection method and device based on point cloud data. First, the original point cloud data and each initial target detection frame output by the initial target detection network are obtained, and each initial target detection frame information is obtained; in order to facilitate subsequent processing The neural network obtains the information of the initial target detection frame. This application extracts the point cloud within the preset frame range around each initial target detection frame from the original point cloud data, and based on the points in the original point cloud data within each initial target detection frame , the points in the point cloud within the preset frame outside the initial target detection frame and the information of each initial target detection frame generate the input data of the neural network. The input data is the data that takes into account the initial target detection frame information, so that the input data is input into the pre-trained neural network for processing, and the detection frame results corresponding to each initial target detection frame are obtained based on the output of the pre-trained neural network. Target category results. In this way, the obtained detection frame results and target category results are more accurate, which can improve the performance of target detection.

Description of drawings

In order to explain the embodiments of the present application or the technical solutions in the prior art more clearly, the drawings needed to be used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings in the following description are only These are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without exerting any creative effort.

Figure 1 is a flow chart 1 of a target detection method based on point cloud data provided by an embodiment of the present application;

Figure 2 is a flow chart 2 of a target detection method based on point cloud data provided by an embodiment of the present application;

Figure 3 is a schematic diagram of the situation where the position of the initial target detection frame is inaccurate in the embodiment of the present application;

Figure 4 is a schematic diagram of the situation where the size of the initial target detection frame is inaccurate in the embodiment of the present application;

Figure 5 is a schematic diagram of the initial target detection frame, preset frame range and virtual points in the embodiment of the present application;

Figure 6 is a schematic structural diagram of a target detection device based on point cloud data provided by an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of protection of this application.

It should be noted that the terms "first", "second", etc. in the description and claims of this application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that data so used may be interchanged where appropriate for the embodiments of the application described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus that encompasses a series of steps or units and need not be limited to those explicitly listed. Those steps or elements may instead include other steps or elements not expressly listed or inherent to the process, method, product or apparatus.

In order to enable those skilled in the art to better understand this application, some technical terms appearing in the embodiments of this application are explained as follows:

Movable objects: refers to vehicles, mobile robots, aircraft and other movable objects. Movable objects can be equipped with various types of sensors, such as lidar, cameras, etc.

Point cloud: Data of the surrounding environment collected through lidar, marked with sparse three-dimensional space points.

Frame: The measurement data received by the sensor after completing an observation. For example, a frame of data from a camera is a picture, and a frame of data from a lidar is a set of laser point clouds.

PointNet: There is a new deep learning model for processing point cloud data, which can complete the processing of the original point cloud data without encoding the original point cloud data.

PointNet++: an improved neural network model of the existing PointNet network.

SECOND: Sparsely Embedded Convolutional Detection, an existing sparsely embedded convolutional target detection network.

PointPillars: A new existing point cloud encoder and network.

MV3D: Multi-View 3D, an existing multi-view three-dimensional network.

PV-RCNN: Point Voxel-RCNN, an existing point voxel integration network.

PointRCNN: A new existing 3D object detection network.

DGCNN: Dynamic Graph Convolutional Neural Network, an existing dynamic graph convolutional neural network.

RSCNN: Relation-Shape CNN, an existing convolutional neural network based on geometric relationships.

NMS: Non Maximum Suppression, non-maximum suppression, as the name suggests, suppresses elements that are not maximum values and searches for local maximum values.

In some embodiments of the present application, the term "vehicle" is interpreted broadly to include any movable object, including, for example, aircraft, boats, spacecraft, cars, trucks, vans, semi-trailers, motorcycles, golf carts, Off-road vehicles, warehouse transport vehicles or agricultural vehicles as well as transport vehicles running on rails, such as trams or trains and other rail vehicles. "Vehicle" in this application may generally include: power systems, sensor systems, control systems, peripherals, and computer systems. In other embodiments, the vehicle may include more, fewer, or different systems.

Among them, the power system is the system that provides power for the vehicle, including: engine/motor, transmission and wheels/tires, and energy unit.

The control system may include a combination of devices that control the vehicle and its components, such as steering units, throttles, braking units.

Peripheral devices may be devices that allow the vehicle to interact with external sensors, other vehicles, external computing devices, and/or users, such as wireless communication systems, touch screens, microphones, and/or speakers.

Based on the vehicle described above, the autonomous vehicle is also equipped with a sensor system and an autonomous driving control device.

The sensor system may include a plurality of sensors for sensing information about the environment in which the vehicle is located, and one or more actuators that change the position and/or orientation of the sensors. The sensor system may include any combination of sensors such as GPS sensors, inertial measurement units, radio detection and ranging (RADAR) units, cameras, laser rangefinders, light detection and ranging (LIDAR) units, and/or acoustic sensors; The sensor system may also include sensors that monitor the vehicle's internal systems (e.g., _O2 monitor, fuel gauge, engine temperature gauge, etc.).

The automatic driving control device may include a processor and a memory. At least one machine-executable instruction is stored in the memory. The processor executes at least one machine-executable instruction to implement a map engine, a positioning module, a perception module, a navigation or path module, and an automatic driving control device. Functions of control modules, etc. The map engine and positioning module are used to provide map information and positioning information. The perception module is used to perceive things in the environment of the vehicle based on the information obtained by the sensor system and the map information provided by the map engine. The navigation or path module is used to plan a driving path for the vehicle based on the processing results of the map engine, positioning module and perception module. The automatic control module parses and converts the decision-making information input of modules such as navigation or path modules into control command output for the vehicle control system, and implements the vehicle through the vehicle network (such as CAN bus, local area Internet network, multimedia directional system transmission, etc. The internal electronic network system) sends control commands to the corresponding components in the vehicle control system to realize automatic control of the vehicle; the automatic control module can also obtain information about each component in the vehicle through the vehicle network.

Currently, existing technologies include inputting point cloud data into various target detection networks for target detection, such as using Sparsely Embedded Convolutional Detection (SECOND), new point cloud encoders and Network (PointPillars), Multi-View 3D (MV3D for short), Point Voxel-RCNN (PV-RCNN for short) or a new type of three-dimensional object detection network (Point RCNN). However, target detection through the above-mentioned existing target detection networks has the problem of poor detection performance. There are two main reasons:

First, when using SECOND, PointPillars, MV3D or PV-RCNN for target detection, various encodings of point cloud data (such as rasterization or bird's-eye view projection, etc.) are required to make the point cloud messy and disordered. Become orderly to reduce the amount of calculation in the target detection process. However, in the process of various encodings, the position information of the original point cloud data is also lost, resulting in inaccurate position of the target detection frame and a decrease in overall detection performance.

Secondly, when using Point RCNN for target detection, the size of the target detection frame predicted by it can be large or small in some sparse point cloud locations (either a larger size or a smaller size can fit the corresponding point cloud into the frame), so When the subsequent processing network (such as PointNet) cannot clearly know the size of the target detection frame, the positive and negative samples divided by each target detection frame are inaccurate, making the subsequent Non-Maximum Suppression (NMS) ) processing will not be accurate enough and target detection performance will be poor.

The embodiments of this application aim to propose a target detection solution based on point cloud data to improve the performance of target detection.

As shown in Figure 1, this embodiment of the present application provides a target detection method based on point cloud data, including:

Step 101: Obtain the original point cloud data and each initial target detection frame output by the initial target detection network, and obtain the information of each initial target detection frame.

Step 102: Extract point clouds within the preset frame range around each initial target detection frame from the original point cloud data.

Step 103: Generate the input data of the neural network based on the points in the original point cloud data within each initial target detection frame, the points in the point cloud within the preset frame outside the initial target detection frame, and the information on each initial target detection frame.

Step 104: Input the input data into the pre-trained neural network for processing, and obtain the detection frame results and target category results corresponding to each initial target detection frame based on the output of the pre-trained neural network.

In order to enable those skilled in the art to better understand the present application, the embodiments of the present application will be described in more detail below with reference to the accompanying drawings, examples, etc. It is worth noting that target detection in the embodiments of the present application may refer to autonomous vehicles, intelligent robots, drones, etc. equipped with radars perceiving the surrounding environment and detecting and identifying objects of concern (such as vehicles, pedestrians, obstacles, etc.) objects, etc., but not limited to this), it can also refer to security facilities equipped with radar to sense the monitored range and detect and identify objects of concern (people, vehicles, containers, etc., but not limited to this). Of course, there are many scenarios that are applied to target detection. It should be noted that all scenarios in which target detection can be applied may be applied to the embodiments of this application, and the embodiments of this application will not be listed one by one here.

In an embodiment of the present application, as shown in Figure 2, a target detection method based on point cloud data is provided, including:

Step 201: Obtain the original point cloud data and each initial target detection frame output by the initial target detection network, and obtain the information of each initial target detection frame.

Here, in an embodiment of the present application, the original point cloud data obtained refers to the original point cloud data collected by radar (such as lidar). In the embodiment of the present application, the original point cloud data will be used for target detection later, without the need to perform various encodings (such as rasterization or bird's-eye view projection, etc.) on these original point cloud data to avoid the need for various encodings. During the process, the position information of the original point cloud data is lost, resulting in inaccurate position of the initial target detection frame and a decrease in overall detection performance.

Here, in an embodiment of the present application, the initial target detection network can be some currently existing target detection networks, such as some that require various encodings of original point cloud data (such as rasterization or bird's-eye view projection, etc. Method) target detection network, such as SECOND, PointPillars, MV3D or PV-RCNN and other target detection networks. The specific working processes of these target detection networks belong to existing technologies and will not be described again here. What needs to be known is that the initial target detection frames output by these target detection networks may have inaccurate positions. For example, in Figure 3, a set of point clouds 31 (such as the external contour point cloud of a vehicle) should ideally be framed by an accurate initial target detection frame, and basically no points fall outside the initial target detection frame. However, the above target detection networks The output initial target detection frame 32 may be as shown in FIG. 3 , and its position may be inaccurate, causing some points in the point cloud 31 to be outside the initial target detection frame 32 . One of the purposes of the target detection method based on point cloud data provided by the embodiment of the present application is to overcome the problem shown in Figure 3 and make the result position of the detection frame more accurate.

In addition, in an embodiment of the present application, the initial target detection network can be some currently existing target detection networks, such as some target detection networks that directly process original point cloud data, such as PointRCNN and other target detection networks. The specific working process of the target detection network belongs to the existing technology and will not be described again here. What needs to be known is that the initial target detection frame output by these target detection networks may have unclear size. If the initial target detection frame is larger or smaller, the corresponding point cloud can be framed, so that subsequent processing can When the network (such as PointNet) cannot clearly know the size of the initial target detection frame, the positive and negative samples divided by each initial target detection frame are inaccurate, making the subsequent Non-Maximum Suppression (NMS) Processing will be less accurate and target detection performance will be poorer. For example, in Figure 4, a set of point clouds 41 (such as the vehicle's outer contour point cloud) should ideally be framed by the initial target detection frame, and the size of the initial target detection frame should be just enough to frame these point clouds 41. However, the initial target detection frame 42 output by the above-mentioned target detection networks may be as shown in FIG. 4 , and may be too large in size, resulting in that although the point cloud 41 is framed, there are too many blank parts in the initial target detection frame 42 . Another purpose of the target detection method based on point cloud data provided by the embodiment of the present application is to overcome the problem shown in Figure 4, so that the detection frame result can more accurately frame the target, and the detection frame size is appropriate.

In addition, since the target detection network can detect multiple objects at the same time, and even for target detection of the same object, there may be multiple initial target detection frames. In order to ensure the subsequent calculation process, in an embodiment of the present application, it is necessary to obtain each initial target detection frame information, where the initial target detection frame information may include the size range information of the initial target detection frame, for example, in the form of (l, w, h ) form, but is not limited to this; among them, l, w and h respectively represent the length, width and height of the initial target detection frame. In addition, the initial target detection frame information may also include initial target detection frame center point information and orientation information. For example, the initial target detection frame center point information is represented by (Cx, Cy, Cz), and Cx, Cy, Cz respectively represent initial target detection. The coordinates of the center point of the box, and the orientation information can be recorded as heading, but it is not limited to this.

Step 202: Extract point clouds within the preset frame range around each initial target detection frame from the original point cloud data.

In an embodiment of the present application, because the initial target detection frame may have some points in the point cloud 31 falling outside the initial target detection frame 32 as shown in Figure 3, in order to meet the processing requirements of the original point cloud data , to avoid missing key points in the point cloud. This step 202 requires extracting the point cloud within the preset frame range around each initial target detection frame from the original point cloud data. The specific process can be used in the following two ways (but is not limited to this):

Method 1: Obtain the preset frame magnification, that is, as shown in Figure 5. The frame magnification refers to the ratio of the preset range that needs to be expanded to (for example, the frame range 51) and the size of the initial target detection frame 52 (such as It is the ratio of the two volumes, the ratio of the two frame lengths, or the ratio of the two surface areas, but is not limited to this.). Generally, the frame magnification is greater than 1, for example, 1.5 times can be selected, but it is not limited to this.

According to each initial target detection frame and frame magnification, determine the range of each preset frame (that is, when the initial target detection frame 52 and frame magnification are known, the preset frame range 51 can be calculated), and from the original point cloud Point clouds within each preset frame are extracted from the data, that is, point clouds outside the initial target detection frame and within the preset frame will be extracted.

Method 2: Obtain a preset frame expansion amount; wherein, the frame expansion amount can be a preset expansion length, expansion area, or expansion volume; for example, as shown in Figure 5, the initial target detection frame 52 has a frame length, a surface area, a frame length Volume and other characteristics, so after obtaining the corresponding frame enlargement, the preset frame range 51 can be determined, and the point clouds within each preset frame range can be extracted from the original point cloud data. That is, point clouds outside the initial target detection frame and within the preset frame will be extracted.

It is worth noting that in the above step 103, the input data of the neural network in the embodiment of the present application may include the point data to be processed and the expression information of each initial target detection frame; the initial target detection frame information and the expression information at least include the initial The size range information of the target detection frame. In addition, the initial target detection frame information and expression information can also include the center point information and orientation information of the initial target detection frame; the point data to be processed includes the point position to be processed (for example, the coordinates can be used Way).

The expression information here may refer to any method that can express the initial target detection frame, which is not limited in this application. For ease of understanding, several common methods will be provided in subsequent embodiments of this application.

After step 202, continue to perform the following steps 203 to 204. This step 203 to step 204 can be used as a specific implementation manner of the above step 103.

Step 203: Determine each initial target detection based on the points in the original point cloud data within each initial target detection frame, the points in the point cloud within the preset frame range outside the initial target detection frame, and the size range information of each initial target detection frame. The expression information of the box.

Here, step 203 can be implemented in a variety of ways. The embodiments of this application only list some of the embodiments. It should be noted that any expression information that can express the initial target detection frame should be understood as the expression information in the embodiments of this application. express information.

Among them, method A can adopt the method of setting virtual points in the initial target detection frame:

For example, according to the size range information of each initial target detection frame, virtual points uniformly filled in each initial target detection frame are generated in each initial target detection frame as expression information of each initial target detection frame. The distribution of this virtual point expresses the range of the initial target detection frame.

Specifically, the interval of virtual points corresponding to each initial target detection frame can be obtained based on the size range information of each initial target detection frame; and then based on the interval of virtual points corresponding to each initial target detection frame, generate in each initial target detection frame The virtual points uniformly filled in the initial target detection frame serve as the expression information of each initial target detection frame. As shown in FIG. 5 , virtual points 53 are evenly distributed within the initial target detection frame 52 . Among them, the interval of the virtual points 53 can be determined according to the size range of the initial target detection frame. For example, the size range of the initial target detection frame is a frame with a length, width and height of 5m, 3m, and 2m respectively, then it can be within the frame range. The center of each cubic decimeter or the eight vertices of each cubic decimeter is determined to set the virtual point 53, but it is not limited to this. Those skilled in the art can also use other methods to obtain the interval of the virtual points, such as directly setting the interval manually.

In order to facilitate the subsequent neural network to identify virtual points, points in the original point cloud data within the initial target detection frame, and points in the point cloud within the preset frame outside the initial target detection frame, these points need to be assigned values, such as : As shown in Figure 5, pre-allocated points are respectively allocated to the points 521 and virtual points 53 in the original point cloud data within each initial target detection frame 52 and the points 511 in the point cloud within the preset frame range 51 outside the initial target detection frame 52. Set the point type value, for example, the point type value corresponding to the point in the original point cloud data within the initial target detection frame is 1, the point type value corresponding to the virtual point is 2, and the point cloud within the preset frame range outside the initial target detection frame The point type value corresponding to the point in is 0, but it is not limited to this.

In addition, method B is a method of adding size range information to the features of points in the point cloud:

For example, the size range information of each initial target detection frame can be added to the characteristics of points in the original point cloud data within the corresponding initial target detection frame and points in the point cloud within the preset frame range outside the initial target detection frame, so as to The expression information is carried in the features, so that after these features are input into the subsequent neural network, the subsequent neural network will be able to obtain expression information that can express the initial target detection frame.

Here, specifically how to add the size range information of the initial target detection frame to the characteristics of the points in the original point cloud data within the corresponding initial target detection frame and the points in the point cloud within the preset frame range outside the initial target detection frame There are many ways, only two of them are listed here, but they are not limited to these two ways.

One of them: you can add the size range information (l, w, h) of each initial target detection frame to the points in the original point cloud data within the corresponding initial target detection frame and the points within the preset frame range outside the initial target detection frame. From the coordinates (xi, yi, zi) of the points in the point cloud, the characteristics (xi, yi, zi, l, w, h) of each point with the expression information are generated; where l, w and h represent respectively The length, width and height of the initial object detection frame.

Another method: based on the size range information (l, w, h) of each initial target detection frame, the points in the original point cloud data within the corresponding initial target detection frame and the points within the preset frame range outside the initial target detection frame can be compared The points (xi, yi, zi) in the cloud are normalized to generate features with the expression information for each point. Among them, l, w and h represent the length, width and height of the initial target detection frame respectively.

Step 204: Generate input data of the neural network that at least contains the position of each point to be processed and the expression information of each initial target detection frame.

Among them, the position of each point to be processed can be in the form of coordinates. The points to be processed generally refer to the points in the original point cloud data within the initial target detection frame and the points in the point cloud within the preset frame range outside the initial target detection frame. .

Step 205: Input the input data into the pre-trained neural network for processing, and obtain the detection frame results and target category results corresponding to each initial target detection frame based on the output of the pre-trained neural network.

Among them, the pre-trained neural network can use the current mainstream target detection network that can process original point cloud data, such as PointNet network, PointNet++ network, dynamic graph convolutional neural network DGCNN or geometric relationship-based convolutional neural network RSCNN, etc. , but not limited to this. Since these neural networks belong to target detection networks in the prior art, the structures of these neural networks will not be described again here. The embodiments of this application only provide input data to these neural networks and obtain the outputs of these neural networks.

Specifically, when the above step 203 adopts method A, the pre-training process of the neural network here is:

Obtain a training sample data set; the training sample data set includes several sets of training sample data; each set of training sample data includes points in the original point cloud data within the initial target detection frame, virtual points within the initial target detection frame, and the initial target detection frame Points in the point cloud within the outer preset frame, point type values corresponding to each of the above points, detection frame results and target category results corresponding to the pre-marked initial target detection frame.

The points in the original point cloud data within the initial target detection frame, the virtual points within the initial target detection frame, the points in the point cloud within the preset frame outside the initial target detection frame, and the point type values corresponding to the above points are used as input , using the detection frame results and target category results corresponding to the pre-marked initial target detection frame as output to train the neural network.

Specifically, when the above step 203 adopts method B, the pre-training process of the neural network here is:

Obtain a training sample data set; the training sample data set includes several sets of training sample data; wherein each set of training sample data includes features of points to be processed within the initial target detection frame with the expression information, and presets outside the initial target detection frame. The points to be processed within the frame range have the characteristics of the expression information, the detection frame results and target category results corresponding to the pre-marked initial target detection frame;

The points to be processed within the initial target detection frame with the characteristics of the expression information, and the points to be processed within the preset frame range outside the initial target detection frame with the characteristics of the expression information are used as input, and the pre-marked initial The detection frame results and target category results corresponding to the target detection frame are used as output to train the neural network.

Here, there are many specific neural network training methods, such as BGD (Batch Gradient Descent, batch gradient descent method), SGD (Stochastic Gradient Descent, stochastic gradient descent method), Adam optimization algorithm (Adam optimization algorithm), RMSprop ( Root Mean Square prop, root mean square prop, etc., but not limited to this.

Correspondingly, when the above step 203 adopts mode A, step 205 here can be implemented in the following way:

The points and virtual points in the original point cloud data within each initial target detection frame, as well as the points in the point cloud within the preset frame outside the initial target detection frame and their corresponding point type values are input into the pre-trained neural network. Process, and obtain the detection frame results and target category results corresponding to each initial target detection frame based on the output of the pre-trained neural network.

Correspondingly, when the above step 203 adopts mode B, step 205 here can be implemented in the following way:

The points to be processed within each initial target detection frame are characterized by the expression information, and the points to be processed within the preset frame outside the initial target detection frame are characterized by the expression information and are input into the pre-trained neural network. Processing is performed in the method, and the detection frame results and target category results corresponding to each initial target detection frame are obtained based on the output of the pre-trained neural network.

After the above steps 201 to 205, the pre-trained neural network processes the original point cloud data without causing any loss of position information of the original point cloud data. In addition, the pre-trained neural network can obtain an expression that represents the initial target detection frame. information, so its output detection frame results can also avoid being too large. The final detection frame result will be closer to the real situation of the target in terms of position and size.

Specifically, the detection frame result output by the pre-trained neural network may include the center point coordinates of the detection frame output by the neural network, the detection frame size information, and the orientation information corresponding to the detection frame.

Specific target category results can generally be expressed in the form of numbers according to the category of concern. For the same target object, the sum of the numbers representing each category in the corresponding target category result is 1. For example, in the whole frame point cloud data, the categories of interest are pedestrians and vehicles respectively, then the target category results can be expressed as scores, that is: (values identified as pedestrians, values identified as vehicles), where for the Pedestrians may output 5 detection frames, and their target category results may be (0.90, 0.10), (0.87, 0.13), (0.78, 0.22), (0.96, 0.04), (0.89, 0.11) respectively. For this vehicle, it may be Four detection frames are output, and their target category results may be (0.05, 0.95), (0.08, 0.92), (0.20, 0.80), (0.15, 0.85) respectively. Generally speaking, in the field of target detection, other methods can be used to express the detection frame results and target category results, which will not be listed here.

After step 205, step 206 can also be continued:

Step 206: Process the detection frame results and target category results corresponding to each initial target detection frame according to the non-maximum suppression NMS algorithm to generate a final detection frame result after non-maximum suppression.

Since there are several above-mentioned detection frame results and corresponding target category results, especially for the same target, there may be multiple detection frames, and multiple detection frames have overlapping areas. Therefore, non-maximum suppression is required to filter out the best Preferred final detection box results. For example, the above-mentioned pedestrians correspond to 5 detection frames, and the target category results are (0.90, 0.10), (0.87, 0.13), (0.78, 0.22), (0.96, 0.04), (0.89, 0.11). The above-mentioned vehicles correspond to The target category results of the 4 detection frames are (0.05, 0.95), (0.08, 0.92), (0.20, 0.80), (0.15, 0.85) respectively. Through the above non-maximum suppression processing, the final detection frame result may only be the detection frame corresponding to (0.96, 0.04) and the detection frame corresponding to (0.05, 0.95). The specific non-maximum suppression algorithm is relatively mature and will not be described again here.

In addition, as shown in Figure 6, this embodiment of the present application also provides a target detection device based on point cloud data, including:

The initial information obtaining unit 61 is used to obtain the original point cloud data and each initial target detection frame output by the initial target detection network, and obtain the information of each initial target detection frame.

The point cloud extraction unit 62 is configured to extract point clouds within the preset frame range around each initial target detection frame from the original point cloud data.

The input data generation unit 63 is used to generate a neural network based on the points in the original point cloud data within each initial target detection frame, the points in the point cloud within the preset frame range outside the initial target detection frame, and the information on each initial target detection frame. input data.

The result generation unit 64 is used to input the input data into a pre-trained neural network for processing, and obtain the detection frame results and target category results corresponding to each initial target detection frame based on the output of the pre-trained neural network.

It is worth noting that the specific implementation of the target detection device based on point cloud data provided by the embodiments of the present application can be referred to the method embodiments corresponding to the above-mentioned Figures 1 and 5, and will not be described again here.

In addition, embodiments of the present application also provide a computer-readable storage medium, including a program or instructions. When the program or instructions are run on a computer, the methods corresponding to the above-mentioned Figures 1 and 5 are implemented.

In addition, embodiments of the present application also provide a computer program product containing instructions, which when the computer program product is run on a computer, causes the computer to execute the method corresponding to the above-mentioned FIG. 1 and FIG. 5 .

In addition, embodiments of the present application also provide a chip system, including a processor, the processor is coupled to a memory, the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the implementation The methods corresponding to the above figures 1 and 5.

In addition, embodiments of the present application also provide a computer server, including a memory, and one or more processors communicatively connected to the memory;

The memory stores instructions that can be executed by the one or more processors, and the instructions are executed by the one or more processors, so that the one or more processors implement the above-mentioned Figure 1 and The method corresponding to Figure 5.

Embodiments of the present application provide a target detection method and device based on point cloud data. First, the original point cloud data and each initial target detection frame output by the initial target detection network are obtained, and each initial target detection frame information is obtained; in order to facilitate subsequent processing The neural network obtains the information of the initial target detection frame. This application extracts the point cloud within the preset frame range around each initial target detection frame from the original point cloud data, and based on the points in the original point cloud data within each initial target detection frame , the points in the point cloud within the preset frame outside the initial target detection frame and the information of each initial target detection frame generate the input data of the neural network. The input data is the data that takes into account the initial target detection frame information, so that the input data is input into the pre-trained neural network for processing, and the detection frame results corresponding to each initial target detection frame are obtained based on the output of the pre-trained neural network. Target category results. In this way, the obtained detection frame results and target category results are more accurate, which can improve the performance of target detection.

Those skilled in the art will understand that embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment that combines software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It will be understood that each process and/or block in the flowchart illustrations and/or block diagrams, and combinations of processes and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device produce a use A device for realizing the functions specified in one process or multiple processes of the flowchart and/or one block or multiple blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory that causes a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction means, the instructions The device implements the functions specified in a process or processes of the flowchart and/or a block or blocks of the block diagram.

These computer program instructions may also be loaded onto a computer or other programmable data processing device, causing a series of operating steps to be performed on the computer or other programmable device to produce computer-implemented processing, thereby executing on the computer or other programmable device. Instructions provide steps for implementing the functions specified in a process or processes of a flowchart diagram and/or a block or blocks of a block diagram.

Specific embodiments are used in this application to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, based on this application The idea of the application may change in the specific implementation mode and application scope. In summary, the contents of this specification should not be understood as limiting the application.

Claims (20)

1. The target detection method based on the point cloud data is characterized by comprising the following steps of:

obtaining original point cloud data and initial target detection frames output by an initial target detection network, and obtaining information of the initial target detection frames, wherein the initial target detection frames are obtained by inputting the original point cloud data into the initial target detection network, one part of points of the original point cloud data fall inside the initial target detection frames, and the other part of points fall outside the initial target detection frames;

extracting point clouds in a preset frame range around each initial target detection frame from the original point cloud data, wherein each preset frame range is larger than and surrounds the corresponding initial target detection frame;

generating input data of a neural network according to points in original point cloud data in each initial target detection frame, points in point cloud within a preset frame range outside the initial target detection frame and information of each initial target detection frame;

and inputting the input data into a pre-trained neural network for processing, and obtaining a detection frame result and a target class result corresponding to each initial target detection frame according to the output of the pre-trained neural network.

2. The method of claim 1, wherein extracting a point cloud within a predetermined frame range around each initial target detection frame from the raw point cloud data comprises:

obtaining a preset frame multiplying power, wherein the frame multiplying power is larger than 1;

and determining each preset frame range according to each initial target detection frame and the frame multiplying power, and extracting point clouds in each preset frame range from the original point cloud data.

3. The method of claim 1, wherein extracting a point cloud within a predetermined frame range around each initial target detection frame from the raw point cloud data comprises:

obtaining a preset frame expansion amount; the frame expansion amount is preset expansion length, expansion area or expansion volume;

and determining each preset frame range according to each initial target detection frame and the frame expansion amount, and extracting point clouds in each preset frame range from the original point cloud data.

4. The method according to claim 1, wherein the input data of the neural network includes point data to be processed and expression information of each initial target detection frame; the initial target detection frame information and the expression information at least comprise size range information of an initial target detection frame; the point data to be processed comprises point positions to be processed;

The generating the input data of the neural network according to the points in the original point cloud data in each initial target detection frame, the points in the point cloud within the preset frame range outside the initial target detection frame and the information of each initial target detection frame comprises the following steps:

determining the expression information of each initial target detection frame according to the points in the original point cloud data in each initial target detection frame, the points in the point cloud within the range of the preset frame outside the initial target detection frame and the size range information of each initial target detection frame;

and generating input data of the neural network at least comprising the expression information of each point position to be processed and each initial target detection frame.

5. The method of claim 4, wherein the initial target detection frame information and the expression information further comprise initial target detection frame center point information and orientation information.

6. The method according to claim 4, wherein determining the expression information of each initial target detection frame based on the points in the original point cloud data in each initial target detection frame, the points in the point cloud within the preset frame range outside the initial target detection frame, and the size range information of each initial target detection frame includes:

Generating virtual points uniformly filled in each initial target detection frame as expression information of each initial target detection frame according to the size range information of each initial target detection frame;

the method further comprises the steps of:

and respectively distributing preset point type values to points in original point cloud data in each initial target detection frame, the virtual points and points in point clouds within a preset frame range outside the initial target detection frame.

7. The method of claim 6, wherein generating virtual points uniformly filled in each initial target detection frame as expression information of each initial target detection frame according to the size range information of each initial target detection frame, comprises:

obtaining the intervals of virtual points corresponding to the initial target detection frames according to the size range information of the initial target detection frames;

and generating virtual points uniformly filled in the initial target detection frames in each initial target detection frame as expression information of each target detection frame according to the intervals of the virtual points corresponding to each initial target detection frame.

8. The method of claim 7, wherein inputting the input data into a pre-trained neural network for processing, obtaining a detection frame result and a target class result corresponding to each initial target detection frame according to an output of the pre-trained neural network, comprises:

And inputting points and virtual points in original point cloud data in each initial target detection frame, points in point clouds within a preset frame range outside the initial target detection frame and respective corresponding point type values into a pre-trained neural network for processing, and obtaining detection frame results and target category results corresponding to each initial target detection frame according to output of the pre-trained neural network.

9. The method of claim 8, further comprising a pre-training process of the neural network:

obtaining a training sample data set; the training sample data set comprises a plurality of groups of training sample data; each group of training sample data comprises points in original point cloud data in an initial target detection frame, virtual points in the initial target detection frame, points in point cloud in a preset frame range outside the initial target detection frame, point type values corresponding to the points, detection frame results corresponding to the initial target detection frame and target type results which are marked in advance;

and taking points in the original point cloud data in the initial target detection frame, virtual points in the initial target detection frame, points in the point cloud within a preset frame range outside the initial target detection frame and point type values corresponding to the points as inputs, taking a detection frame result corresponding to the initial target detection frame and a target class result which are marked in advance as outputs, and training the neural network.

10. The method according to claim 4, wherein determining the expression information of each initial target detection frame based on the points in the original point cloud data in each initial target detection frame, the points in the point cloud within the preset frame range outside the initial target detection frame, and the size range information of each initial target detection frame includes:

and adding the size range information of each initial target detection frame into the corresponding characteristics of the points in the original point cloud data in the initial target detection frame and the points in the point cloud within the range of the preset frame outside the initial target detection frame, so that the characteristics carry the expression information.

11. The method according to claim 10, wherein adding the size range information of each initial target detection frame to the corresponding feature of the point in the original point cloud data in the initial target detection frame and the point in the point cloud within the preset frame range outside the initial target detection frame, so that the feature carries the expression information, includes:

size range information of each initial target detection framel、w、h) Coordinates of points in original point cloud data added to corresponding initial target detection frame and points in point cloud within a preset frame range outside the initial target detection frame xi、yi、zi) In the process, the characteristics that each point has the expression information are generatedxi、yi、zi、l、w、h) The method comprises the steps of carrying out a first treatment on the surface of the Wherein,l、wandhrepresenting the length, width and height of the initial target detection frame, respectively.

12. The method according to claim 10, wherein adding the size range information of each initial target detection frame to the corresponding feature of the point in the original point cloud data in the initial target detection frame and the point in the point cloud within the preset frame range outside the initial target detection frame, so that the feature carries the expression information, includes:

according to the size range information of each initial target detection framel、w、h) The corresponding points in the original point cloud data in the initial target detection frame and the points in the point cloud within the range of the preset frame outside the initial target detection frame are treated with the methodxi、yi、zi) Normalized processing is carried out to generate the characteristics that each point has the expression information) The method comprises the steps of carrying out a first treatment on the surface of the Wherein,l、wandhrepresenting the length, width and height of the initial target detection frame, respectively.

13. The method according to claim 11 or 12, wherein inputting the input data into a pre-trained neural network for processing, obtaining a detection frame result and a target class result corresponding to each initial target detection frame according to an output of the pre-trained neural network, comprises:

And inputting the characteristics of the expression information of the points to be processed in each initial target detection frame and the characteristics of the expression information of the points to be processed in a range of a preset frame outside the initial target detection frame into a pre-trained neural network for processing, and obtaining detection frame results and target class results corresponding to each initial target detection frame according to the output of the pre-trained neural network.

14. The method of claim 13, further comprising a pre-training process of the neural network:

obtaining a training sample data set; the training sample data set comprises a plurality of groups of training sample data; each group of training sample data comprises characteristics of points to be processed in an initial target detection frame with the expression information, characteristics of points to be processed in a preset frame range outside the initial target detection frame with the expression information, detection frame results corresponding to the initial target detection frame and target category results, which are marked in advance;

and taking the characteristics of the expression information of the points to be processed in the initial target detection frame and the characteristics of the expression information of the points to be processed in the range of the preset frame outside the initial target detection frame as inputs, taking the detection frame result and the target class result corresponding to the initial target detection frame which are marked in advance as outputs, and training the neural network.

15. The method of claim 1, wherein the detection frame result includes center point coordinates of a detection frame output by the neural network, detection frame size information, and orientation information corresponding to the detection frame.

16. The method of claim 1, further comprising, after obtaining the detection frame result and the target class result corresponding to each initial target detection frame:

and processing the detection frame results and the target class results corresponding to the initial target detection frames according to a non-maximum suppression NMS algorithm to generate final detection frame results after non-maximum suppression.

17. A point cloud data-based object detection apparatus, comprising:

an initial information obtaining unit configured to obtain initial point cloud data and initial target detection frames output by an initial target detection network, and obtain initial target detection frame information, where each initial target detection frame is obtained by inputting the initial point cloud data to the initial target detection network, a part of points of the initial point cloud data fall within each initial target detection frame, and another part of points of the initial point cloud data fall outside each initial target detection frame;

The point cloud extraction unit is used for extracting point clouds in a preset frame range around each initial target detection frame from the original point cloud data, wherein each preset frame range is larger than and surrounds the corresponding initial target detection frame;

the input data generating unit is used for generating input data of the neural network according to points in original point cloud data in each initial target detection frame, points in point cloud in a preset frame range outside the initial target detection frame and information of each initial target detection frame;

and the result generating unit is used for inputting the input data into a pre-trained neural network for processing, and obtaining a detection frame result and a target class result corresponding to each initial target detection frame according to the output of the pre-trained neural network.

18. A computer readable storage medium comprising a program or instructions which, when run on a computer, implement the method of any one of claims 1 to 16.

19. A system on a chip comprising a processor coupled to a memory, the memory storing program instructions that when executed by the processor implement the method of any one of claims 1 to 16.

20. A computer server comprising a memory, and one or more processors communicatively coupled to the memory;

stored in the memory are instructions executable by the one or more processors to cause the one or more processors to implement the method of any one of claims 1 to 16.