CN110147106A - Intelligent mobile service robot with laser and visual fusion obstacle avoidance system - Google Patents

Abstract

The invention relates to the field of intelligent mobile service robots, an intelligent mobile service robot with a laser and vision fusion obstacle avoidance system, the laser and vision fusion obstacle avoidance system includes a hardware system and a navigation obstacle avoidance system, the hardware system includes a depth camera, a laser radar, an airborne PC, motion control board and motor driver, navigation obstacle avoidance system includes obstacle positioning and data conversion module, obstacle classification recognition module, data fusion module, laser navigation framework, chassis drive module and chassis motion control module; navigation obstacle avoidance system includes The chassis motion control module runs on the motion control board, and the rest run on the airborne PC; the laser radar, depth camera and motion control board are connected to the airborne PC, and the motor driver is connected to the motion control board. The system is integrated in the two-dimensional laser obstacle avoidance The visual sensor considers the type of obstacle and the three-dimensional information of the obstacle in the obstacle, and improves the accuracy, robustness and environmental adaptability of the intelligent mobile service robot for obstacle avoidance.

Description

Intelligent mobile service robot with laser and visual fusion obstacle avoidance system

technical field

The invention relates to the field of intelligent mobile service robots, in particular to the field of obstacle avoidance systems for intelligent mobile service robots.

Background technique

The intelligent mobile service robot can continuously move autonomously in multiple scenes in real time, interact with the environment and personnel through various sensors such as voice, vision, and touch, and provide corresponding services to replace the roles of lobby managers and waiters. It can be applied to homes, Hotel, bank, airport and other scenes. In order to ensure safety, the service robot needs to avoid obstacles such as tables, chairs, animals, and people in the scene in time during its autonomous movement. The obstacle avoidance method of intelligent mobile service robots generally obtains the position information of external obstacles through sensors, plans a local path on the cost map, and moves along the planned path through the motion control system, so as to avoid obstacles.

At present, the obstacle avoidance methods of service robots mainly include ultrasonic obstacle avoidance and TOF obstacle avoidance. Infrared rays or lasers emit beams of specific wavelengths, record the reflection time difference, and calculate the distance distribution of nearby obstacles; the existing obstacle avoidance methods use single sensors such as ultrasound and two-dimensional lasers. Due to the limitations of a single sensor, only rely on One type of sensor cannot obtain perfect obstacle information; ultrasonic obstacle avoidance has high requirements on the smoothness of the reflective surface, and safety issues cannot be guaranteed when facing obstacles with no reflective ability or weak reflective ability; two-dimensional laser avoidance The classification and recognition of obstacles in the environment cannot be realized, so different obstacle avoidance strategies are adopted for different categories, and because the two-dimensional laser can only capture the surrounding environment information on the same height plane as the laser sensor, so for special-shaped obstacles objects, such as a hollow table, a hollow door at the bottom, etc., its specific location cannot be judged.

Contents of the invention

The purpose of the present invention is to provide a fusion of visual sensors in two-dimensional laser obstacle avoidance, consider the type of obstacle and the three-dimensional information of the obstacle in the barrier, and improve the accuracy, robustness and reliability of the intelligent mobile service robot for obstacle avoidance. A laser vision fusion obstacle avoidance system for an intelligent mobile service robot with environmental adaptability.

In order to achieve the above object, the technical solution of the present invention is: an intelligent mobile service robot with a laser and vision fusion obstacle avoidance system, the laser and vision fusion obstacle avoidance system includes a hardware system and a navigation system arranged on the main body of the mobile service robot. Obstacle avoidance system, the hardware system includes depth camera, laser radar, airborne PC, motion control board and motor driver, and the navigation obstacle avoidance system includes obstacle positioning and data conversion module, obstacle classification and identification module, data fusion module , a laser navigation frame and a chassis motion control module; the chassis motion control module runs on a motion control board, and the obstacle positioning and data conversion module, obstacle classification and identification module, data fusion module and laser navigation frame run on an airborne On the PC; the lidar, depth camera and motion control board are connected to the onboard PC, and the motor driver is connected to the motion control board.

The navigation and obstacle avoidance method of the laser and visual fusion obstacle avoidance system is to send the visual data obtained by the depth camera to the obstacle positioning and data conversion module and the obstacle classification recognition module respectively, and send the laser data obtained by the laser radar to the data center. The fusion module, the obstacle positioning and data conversion module processes the visual data to obtain the positioning information data of the obstacle and sends it to the data fusion module, and the obstacle classification and identification module processes the visual data to obtain the type information of the obstacle The data is sent to the data fusion module, and the data fusion module processes the laser data and the location information data of the obstacle and the type information data of the obstacle to obtain the position information data of the fused obstacle and send it to the laser navigation framework, the said The laser navigation framework processes the position information data of obstacles to obtain a new local cost map, and the laser navigation framework performs local path planning to obtain the local path planning data and sends it to the chassis driving module, and the chassis driving module plans the local path planning After the data is processed, the chassis motion control data is sent to the chassis motion control module, and the chassis motion control module sends a motion control command to the motor driver.

The obstacle classification recognition module utilizes the deep learning algorithm of the convolutional neural network to use the yolo V3 pre-training model to train the obstacles in the specific scene to obtain the trained neural network model, and the trained neural network model can be passed through according to the depth. The color image information data captured in the visual data obtained by the camera is processed and judged to obtain the type information data of the obstacle.

The obstacle positioning and data conversion module includes three-dimensional information detection of obstacles and virtual two-dimensional laser information conversion. The three-dimensional information detection converts the depth image collected by the depth camera into a digital image that can be recognized by the airborne PC through the image acquisition card and performs The image preprocessing obtains the depth information of the three-dimensional image, and the virtual two-dimensional laser information conversion converts the depth information of the three-dimensional image into virtual two-dimensional laser information data through the depthimage_to_laserscan algorithm, and the virtual two-dimensional laser information data is the obstacle Location information data.

The data fusion module fuses the virtual two-dimensional laser information data, the obstacle type information data and the point cloud information data in the laser data through the Kalman filter algorithm to obtain the position information data of the obstacle.

The hardware system also includes an ultrasonic sensor and a mechanical anti-collision sensor, the ultrasonic sensor and the mechanical anti-collision sensor are connected to the chassis motion control module on the motion control board, and the data obtained by the ultrasonic sensor and the mechanical anti-collision sensor are sent to The chassis motion control module, when the chassis motion control module receives the data sensed by the ultrasonic sensor and the mechanical anti-collision sensor as obstacle sensing data, it sends a control command to the motor driver to stop.

The ultrasonic sensor sends obstacle sensing data when the sensing distance is 10-50 cm; the mechanical anti-collision sensor sends obstacle sensing data when a collision occurs.

By adopting the above-mentioned technical scheme, the beneficial effect of the present invention is: the intelligent mobile service robot with the laser and visual fusion obstacle avoidance system of the present invention adopts the above-mentioned system structure, adopts the combination of laser radar and depth camera through the above-mentioned system involved in the present invention The navigation and obstacle avoidance method realizes a kind of laser and vision fusion so as to carry out the navigation and obstacle avoidance method of the local path planning of the laser navigation frame, thereby realizing the accuracy, robustness and environmental protection of the intelligent mobile service robot to improve the obstacle avoidance of the present invention Purpose of Adaptability.

The laser data obtained through the application of the laser radar can obtain accurate environmental orientation and distance information, and establish an accurate environmental map. The data obtained through the application of the depth camera is processed by the obstacle positioning and data conversion module and the obstacle classification and identification module. That is, the deep perception of the surrounding environment is carried out, and the deep learning algorithm is integrated on the intelligent mobile service robot platform, so as to obtain the positioning information data of obstacles, that is, to obtain the target type and three-dimensional position information of surrounding obstacles, that is, the visual three-dimensional information, through By converting the visual three-dimensional position information, the obstacle positioning information data, that is, the laser two-dimensional plane map, is combined with the obstacle type information, and sent to the laser navigation framework for processing to obtain a new local cost map, and then perform local Path planning thus achieves obstacle avoidance navigation of laser and visual fusion, realizes the fusion of visual sensors in two-dimensional laser obstacle avoidance in the present invention, considers the type of obstacle and the three-dimensional information of obstacles in barriers, and improves the obstacle avoidance of intelligent mobile service robots. The purpose of accuracy, robustness and environmental adaptability.

Description of drawings

Fig. 1 and Fig. 2 are the structural representations of the intelligent mobile service robot involved in the present invention;

Fig. 3 is the structural block diagram of the hardware system involved in the present invention;

Fig. 4 is a structural block diagram of the laser and vision fusion obstacle avoidance system involved in the present invention.

In the picture:

Fuselage main body 1; traveling mechanism 2; depth camera 31; laser radar 32;

Airborne PC33; motion control board 34; motor driver 35;

Obstacle location and data conversion module 41; Obstacle classification and recognition module 42;

Data fusion module 43; Laser navigation framework 44;

Chassis motion control module 45; ultrasonic sensor 36; mechanical anti-collision sensor 37.

Detailed ways

In order to further explain the technical solution of the present invention, the present invention will be described in detail below through specific examples.

The intelligent mobile service robot with laser and visual fusion obstacle avoidance system disclosed by the present invention is shown in Figure 1, Figure 2, Figure 3 and Figure 4, and the intelligent mobile service robot in the figure is a fuselage with a shape structure similar to that of a human body Main body 1, the bottom of the fuselage main body 1 is the walking mechanism 2, the difference between the intelligent mobile service robot of the present invention and the existing intelligent mobile service robot is that the intelligent mobile service robot of the invention includes a laser and visual fusion obstacle avoidance system, The system includes a hardware system and a navigation obstacle avoidance system arranged on the main body of the fuselage. Specifically, the hardware system includes a depth camera 31, a laser radar 32, an airborne PC 33, a motion control board 34, and a motor driver 35. The obstacle system includes an obstacle positioning and data conversion module 41, an obstacle classification and recognition module 42, a data fusion module 43, a laser navigation framework 44 and a chassis motion control module 45, and the chassis motion control module 45 runs on the motion control board 34, The obstacle location and data conversion module 41, the obstacle classification recognition module 42, the data fusion module 43 and the laser navigation framework 44 run on the airborne PC33; the laser radar 32, the depth camera 31 and the motion control board 34 are connected to the machine Carry PC33, the connection here can adopt modes such as network port communication, USB communication, serial port communication, signal line to connect, described motor driver 35 connects motion control board 34 to be used for driving traveling mechanism 2, airborne PC33, motion control board 34 and The motor driver 35 is usually arranged inside the fuselage body 1, and the depth camera 31 and the laser radar 32 are arranged on the shell of the fuselage body 1, with the working end facing outward. As shown in the figure, the depth camera 31 and the laser radar 32 32 is arranged on the front side of the human body shape structure of the intelligent mobile service robot. The shell of the main body 1 of the intelligent mobile service robot can also be equipped with display screens and other equipment connected to the on-board PC. Usually the intelligent mobile service robot also has It may include a loudspeaker, a sound receiver, etc. Since these are not the main improvements of the present invention, other component devices will not be described in this specific embodiment.

The navigation and obstacle avoidance method of the laser and visual fusion obstacle avoidance system of the present invention are as follows:

The visual data obtained by depth camera 31 are sent to obstacle location and data conversion module 41 and obstacle classification recognition module 42 respectively; In-depth learning algorithm, relevant information can be found on the Internet, and will not be described in detail here) using yolo V3 (yolo is a known calculation method, its full name is You Only Look Once, yolo V3 is yoloV3 of the yolo series, is The latest algorithm of this series is also a known algorithm. Relevant information can be found on the Internet, so I will not describe it in detail here. In addition, here is a description of the calculation method that can achieve similar or identical algorithm effects to the yolo V3 algorithm. This embodiment only discloses a realizable yolo V3 algorithm suitable for intelligent mobile service robots, and the adoption of any algorithm falls within the scope of protection of the present invention) The pre-training model is trained on obstacles in specific scenarios to obtain A trained neural network model, the trained neural network model can be processed and judged to obtain the type information data of the obstacle according to the color image information data captured in the visual data obtained by the depth camera; the obstacle location And the data conversion module 41 includes the three-dimensional information detection of obstacles and the virtual two-dimensional laser information conversion, and the three-dimensional information detection converts the depth image collected by the depth camera 31 into a digital image that can be recognized by the airborne PC33 through the image acquisition card (existing product) And carry out image preprocessing to obtain the depth information of the three-dimensional image, and convert the virtual two-dimensional laser information through the depthimage_to_laserscan algorithm (a known calculation method, relevant information can be found on the Internet, and will not be described in detail here. In addition, here is an explanation Computing methods that can achieve a similar or identical algorithm effect to the depthimage_to_laserscan algorithm can also be applied. In this embodiment, only one realizable depthimage_to_laserscan algorithm suitable for intelligent mobile service robots is disclosed. The adoption of any algorithm falls under the protection of the present invention. within the range) to convert the depth information of the three-dimensional image into virtual two-dimensional laser information data, and the virtual two-dimensional laser information data is the positioning information data of the obstacle.

The laser data obtained by the laser radar 32 is sent to the data fusion module 43, and the obstacle location and data conversion module 41 processes the visual data to obtain the location information data of the obstacle and sends it to the data fusion module 43, and the obstacle classification The recognition module 42 processes the visual data to obtain the type information data of the obstacle and sends it to the data fusion module 43, and the data fusion module 43 processes the laser data and the location information data of the obstacle and the type information data of the obstacle to obtain The fused position information data of the obstacle is sent to the laser navigation framework 44, here we first explain the laser navigation framework 44.

The laser navigation framework 44 is a known laser navigation technology. This framework usually includes a map service module, a move_base module, a chassis driver module, a tf change module, an amcl positioning module, an odometer module, etc., and the move_base module usually includes a global Cost map, global path planning, local cost map, local path planning, etc., a method of navigation Here is a brief explanation of importing the grid map into the map service module for processing and transferring the map data to the global cost map to navigate the target position It is determined to obtain the global cost map through the global path for global path planning, and the tf change module, amcl positioning module, and odometer module process and calculate the global path planning data to be sent to the chassis driver module through local path planning. When there is a local change in the map , get the local cost map and replace it into the corresponding part of the global cost map, and replace the local path into the global path planning through the local path planning, and then get the path planning data after processing by other modules and send it to the chassis driver module, so as to achieve the navigation function. Lidar is also applied to some existing navigation technologies, but the application of the prior art is for the use of establishing maps in the early stage, but the present invention is different. The present invention combines the application of Lidar and depth camera, which will be described below Embodiments of the present invention.

The data fusion module 43 fuses the point cloud information data in the laser data, the virtual two-dimensional laser information data and the type information data of the obstacle through the Kalman filter algorithm to obtain the position information data of the obstacle. There are other algorithms for the calculation method of data fusion, which can also be applied to the present invention. In this embodiment, only a realizable Kalman filter algorithm suitable for intelligent mobile service robots is disclosed, which can actually be achieved according to the on-site use environment. The adopted algorithm is determined based on the effect of the algorithm, and the adoption of any algorithm falls within the protection scope of the present invention.

The laser navigation framework 44 can process the position information data of obstacles in the same way as disclosed above to obtain a new local cost map and replace it into the corresponding part of the global cost map, and perform calculations such as local path planning to obtain local path planning data. , or replace into the global path planning, the laser navigation framework 44 performs local path planning to obtain the local path planning data (that is, the data to achieve the effect of navigation and obstacle avoidance) and send it to the chassis drive module, and the chassis drive module will plan the local path The data is processed and the chassis motion control data is sent to the chassis motion control module 45, and the chassis motion control module 45 sends a motion control command to the motor driver 35, so as to control the navigation and obstacle avoidance of the intelligent mobile service robot. The visual sensor is integrated in the 3D laser obstacle avoidance, and the type of obstacle and the three-dimensional information of the obstacle are considered in the obstacle, so as to improve the accuracy, robustness and environmental adaptability of the intelligent mobile service robot for obstacle avoidance.

In the present invention, the hardware system can also include an ultrasonic sensor 36 and a mechanical anti-collision sensor 37, the ultrasonic sensor 36 and the mechanical anti-collision sensor 37 are connected to the chassis motion control module on the motion control board 34, the ultrasonic sensor 36 and the mechanical anti-collision sensor 37 The data sensed by the mechanical anti-collision sensor 37 can be directly sent to the chassis motion control module. 35 send control commands to stop, as the sensing distance of the ultrasonic sensor 36 is set to 10-50 centimeters (preferred can be 20 centimeters or set according to actual conditions) when sending obstacle sensing data, so that intelligent movement The service robot stops moving, and the mechanical anti-collision sensor 37 sends obstacle sensing data when a touch or collision occurs, so that the intelligent mobile service robot stops moving.

As shown in the figure, the ultrasonic sensor 36 and the mechanical anti-collision sensor 37 are arranged on the front side of the shape structure of the intelligent mobile service robot similar to the human body shape, as the depth camera 31 and the laser radar 32. As shown in the figure, the Ultrasonic sensor 36, mechanical anti-collision sensor 37, depth camera 31 and laser radar 32 are arranged at intervals from top to bottom on the front side of the human-shaped shape structure of the intelligent mobile service robot. The depth camera 31 is at the top, and the mechanical anti-collision The sensor 37 is at the bottom, and the laser radar 32 and the ultrasonic sensor 36 are between the depth camera 31 and the mechanical anti-collision sensor 37. The arrangement of the ultrasonic sensor 36 and the mechanical anti-collision sensor 37 can further improve the accuracy, robustness and environmental adaptability of the intelligent mobile service robot for obstacle avoidance.

The above-mentioned embodiments and drawings do not limit the form and style of the product of the present invention, and any appropriate changes or modifications made by those skilled in the art should be considered as not departing from the patent scope of the present invention.

Claims (10)

1. An intelligent mobile service robot with a laser and vision fusion obstacle avoidance system, characterized in that: the laser and vision fusion obstacle avoidance system includes a hardware system and a navigation obstacle avoidance system arranged on the main body of the mobile service robot, the The hardware system includes depth camera, laser radar, airborne PC, motion control board and motor driver. The navigation and obstacle avoidance system includes obstacle positioning and data conversion module, obstacle classification and recognition module, data fusion module, laser navigation framework and chassis Motion control module; the chassis motion control module runs on the motion control board, and the obstacle positioning and data conversion module, obstacle classification recognition module, data fusion module and laser navigation framework run on the airborne PC; the laser The radar, the depth camera and the motion control board are connected to the airborne PC, and the motor driver is connected to the motion control board. 2. The intelligent mobile service robot with laser and vision fusion obstacle avoidance system as claimed in claim 1, characterized in that: the navigation and obstacle avoidance method of the laser and vision fusion obstacle avoidance system is the visual data obtained by the depth camera Send them to the obstacle positioning and data conversion module and the obstacle classification and recognition module respectively, and send the laser data obtained by the laser radar to the data fusion module, and the obstacle positioning and data conversion module processes the visual data to obtain the location of the obstacle The information data is sent to the data fusion module, and the obstacle classification and identification module processes the visual data to obtain the type information data of the obstacle and sends it to the data fusion module, and the data fusion module combines the laser data and the positioning information data of the obstacle with The type information data of the obstacle is processed to obtain the fused position information data of the obstacle and sent to the laser navigation framework, and the laser navigation framework processes the position information data of the obstacle to obtain a new local cost map. The navigation frame performs local path planning to obtain the local path planning data and send it to the chassis driving module. The chassis driving module processes the local path planning data to obtain chassis motion control data and sends it to the chassis motion control module. The chassis motion control module sends The motor driver sends motion control commands. 3. The intelligent mobile service robot with laser and vision fusion obstacle avoidance system as claimed in claim 2, characterized in that: said obstacle classification and identification module utilizes the pre-training model calculation method through the deep learning algorithm of convolutional neural network to Obstacles in a specific scene are trained to obtain a trained neural network model, and the trained neural network model can be processed and judged according to the color image information data captured in the visual data obtained through the depth camera to obtain the type of the obstacle information data. 4. The intelligent mobile service robot with laser and visual fusion obstacle avoidance system according to claim 3, characterized in that: the obstacle positioning and data conversion module includes three-dimensional information detection of obstacles and virtual two-dimensional laser information conversion , 3D information detection converts the depth image collected by the depth camera into a digital image that can be recognized by the airborne PC through the image acquisition card, and performs image preprocessing to obtain the depth information of the 3D image. The virtual 2D laser information conversion converts the 3D image through calculation methods The depth information of the object is converted into virtual two-dimensional laser information data, and the virtual two-dimensional laser information data is the positioning information data of the obstacle. 5. The intelligent mobile service robot with laser and visual fusion obstacle avoidance system as claimed in claim 4, characterized in that: said data fusion module fuses point cloud information data and virtual two-dimensional laser information in laser data by computing methods The data and the type information data of the obstacle are used to obtain the position information data of the obstacle. 6. The intelligent mobile service robot with laser and visual fusion obstacle avoidance system according to claim 2, characterized in that: the obstacle positioning and data conversion module includes three-dimensional information detection of obstacles and virtual two-dimensional laser information conversion , 3D information detection converts the depth image collected by the depth camera into a digital image that can be recognized by the airborne PC through the image acquisition card, and performs image preprocessing to obtain the depth information of the 3D image. The virtual 2D laser information conversion converts the 3D image through calculation methods The depth information of the object is converted into virtual two-dimensional laser information data, and the virtual two-dimensional laser information data is the positioning information data of the obstacle. 7. The intelligent mobile service robot with laser and visual fusion obstacle avoidance system as claimed in claim 2, characterized in that: said data fusion module fuses point cloud information data and virtual two-dimensional laser information in laser data by calculation methods The data and the type information data of the obstacle are used to obtain the position information data of the obstacle. 8. The intelligent mobile service robot with laser and visual fusion obstacle avoidance system according to claim 3, characterized in that: said data fusion module fuses point cloud information data and virtual two-dimensional laser information in laser data by calculation methods The data and the type information data of the obstacle are used to obtain the position information data of the obstacle. 9. The intelligent mobile service robot with laser and visual fusion obstacle avoidance system according to any one of claims 1-8, characterized in that: the hardware system also includes an ultrasonic sensor and a mechanical anti-collision sensor, and the ultrasonic sensor Connect the chassis motion control module on the motion control board with the mechanical anti-collision sensor, and send the data obtained by the ultrasonic sensor and the mechanical anti-collision sensor to the chassis motion control module, and the chassis motion control module receives the ultrasonic sensor and the mechanical anti-collision sensor. When the data obtained by the collision sensor is the obstacle sensing data, the control command is sent to the motor driver to stop. 10. The intelligent mobile service robot with laser and vision fusion obstacle avoidance system as claimed in claim 5, characterized in that: the pre-training model method is a yolo V3 pre-training model calculation method; the depth information of the three-dimensional image The calculation method for converting into virtual two-dimensional laser information data is depthimage_to_laserscan algorithm; the calculation method for point cloud information data, virtual two-dimensional laser information data and obstacle type information data in the fusion laser data is Kalman filter algorithm.