WO2018137445A1

WO2018137445A1 - Ros-based mechanical arm grabbing method and system

Info

Publication number: WO2018137445A1
Application number: PCT/CN2017/117168
Authority: WO
Inventors: 张光肖
Original assignee: 南京阿凡达机器人科技有限公司
Priority date: 2017-01-25
Filing date: 2017-12-19
Publication date: 2018-08-02
Also published as: CN106826822B; CN106826822A

Abstract

A ROS-based mechanical arm grabbing method and system. An implementation process comprises: configuring a use environment of a camera (20) on an upper computer (10), and then disposing the camera (20) above or laterally above an object to be grabbed and obtaining an image of the object to be grabbed that contains a positioning mark; inputting the image to the upper computer (10); reading the image of the camera (20) and performing data processing by using a particular algorithm to obtain spatial pose information of the object to be grabbed at mechanical arm coordinates; and giving a motion information queue and sending same to a lower computer (30). The lower computer (30) receives and parses the motion information queue sent by the upper computer (10), and drives a mechanical arm to perform a grabbing operation according to a preset action. The ROS-based mechanical arm grabbing method and system can effectively utilize the powerful processing operation capability of the upper computer (10), readily implement layout of the mechanical arm and the upper computer (10) and collaborative operation of multiple mechanical arms, and easily implement motion planning of the mechanical arms by using a ROS, and thus have a wide application prospect.

Description

Robot arm grabbing method and system based on ROS system

This application claims the priority of the Chinese patent application filed on January 25, 2017, filed on Jan. 25, 2017, entitled "A ROS-based Visual Positioning and Robotic Arm Crawling Implementation Method", the entire contents of which are incorporated in this.

Technical field

The invention belongs to the field of mechanical arm control and motion planning, in particular to a robot arm grasping method and system based on ROS system.

Background technique

The robotic arm is one of the most widely used automation devices in the field of robotics. In particular, multi-degree-of-freedom manipulators play an increasing role in many fields such as machine building, automotive, semiconductor, medical, and home services. Motion control has always been a hot topic of research. At present, the main application scenarios of the robot arm are as follows:

1) Welding field. Used to perform welding in a poor welding environment instead of manually.

2) The field of automated production lines. It is mainly used to perform actions such as grabbing, flipping, sorting items, etc., to improve production efficiency.

3) The medical field. Mainly used to perform sophisticated medical procedures such as minimally invasive surgery.

4) Service area. In conjunction with mobile robots, the robotic arm has already entered life to perform tasks such as picking up items and picking up debris.

Machine vision belongs to a branch of artificial intelligence. In short, machine vision is to use the camera instead of the human eye to judge and analyze the surrounding environment, combined with certain algorithms to achieve intelligent decision making. It is a comprehensive technology, including image processing. , mechanical engineering technology, control, electric light source illumination, optical imaging, sensors, analog and digital video technology, computer hardware and software technology. Machine vision is divided into several types, such as monocular, binocular, and 3D vision. Its introduction has the following advantages:

1) Machine vision is more reliable than the human eye. Machine vision continuously captures images and works continuously without visual fatigue.

2) Machine vision has higher precision. With certain processing algorithms, machine vision can achieve accurate measurement and error checking, and is conducive to data recording and integration.

3) Machine vision can adapt to complex environments. In some situations that are not suitable for manual work, machine vision can be “out of the box”.

The ROS system (Robot Operating System) is an open source robot operating system released by Willow Garage in 2010. It adopts a distributed organizational structure, which can greatly improve the reusability of code and the adaptability of complex robot systems. The ROS system has the following main features:

1) Point-to-point distributed design. The peer-to-peer design of ROS and the mechanisms such as services and node managers can decentralize the real-time computational pressure brought by functions such as computer vision and speech recognition, and can adapt to the challenges faced by multiple robots.

2) Multi-language support. The ROS system supports programming languages such as C++, Python, Octave, and LISP, as well as interfaces to other programming languages.

3) The software package is rich. The ROS system integrates a large number of software packages, which can quickly realize the environment configuration of various applications of robots, such as robot arm motion planning, mobile robot navigation, robot SLAM and so on.

4) Open source and free. The open source nature of the ROS system encourages more people to contribute their work.

Due to the limitations of the conditions, the intelligence of the robot arm is still not high enough, and more is to perform mechanical teaching actions. There are still many problems for the changing environment.

Summary of the invention

In view of the defects or deficiencies of the prior art, the present invention aims to propose a method and system for grasping a mechanical arm based on the ROS system, which can effectively solve the problems of poor adaptability of the mechanical arm environment and high difficulty in development and use. The present invention provides a complete solution for visual access, target detection, image processing, robotic arm motion planning, etc. of the robot arm.

The technical solution to achieve the object of the present invention is:

A mechanical arm grabbing method based on ROS system, comprising:

Step 2: The upper computer obtains an image of the object to be grasped containing the positioning mark through the camera;

Step 3: The host computer processes the acquired image under the ROS system, and obtains Spatial pose information of the object to be grasped in a robot arm coordinate system;

Step 4: The upper computer performs motion planning on the mechanical arm under the ROS system according to the spatial pose information of the object to be grasped in the robot arm coordinate system and the spatial pose information of the mechanical arm, and obtains corresponding Motion information queue;

Step 5: The host computer transmits the obtained motion information queue of the robot arm to the lower position machine;

Step 6: The lower computer drives the robot arm according to the motion information queue to perform a grab operation according to a corresponding path.

Further, the step 1 before the host computer configures the camera to use the environment under the ROS system further includes: Step 0: The host computer performs positioning mark training on the camera.

Further, the process of performing the positioning mark training on the camera by the upper computer includes the following step: Step 01: The upper computer performs the training of the positioning mark on the camera based on the ARToolKit positioning mark recognition algorithm; Step 02 The upper computer trains the positioning mark on the camera based on the positioning mark recognition algorithm of OpenCV_ArUco.

Further, the step 2: before the upper computer obtains the image of the object to be grasped including the positioning mark by the camera, the method further includes: Step 11: The host computer configures the camera node to drive the camera under the ROS system; Step 12: The upper computer calibrates the camera under the ROS system and saves the correction data; Step 13: The upper computer selects a positioning mark recognition algorithm.

Further, in the step 3, the upper computer processes the acquired image under the ROS system, and the specific process of obtaining the spatial pose information of the object to be grasped in the robot arm coordinate system with the positioning mark is: Step 31: Search for the positioning mark with the highest matching degree with the preset positioning mark in the image acquired by the camera; Step 32: Locating the found positioning mark; Step 33: Obtain the above according to the positioned positioning mark Spatial pose information of the object to be grasped with the positioning mark in the camera coordinate system; Step 34: Converting the matrix according to the preset camera coordinate system and the robot arm coordinate system, and the object to be grasped is in the camera coordinate system The spatial pose information is converted to obtain spatial pose information of the object to be grasped in a robot arm coordinate system.

Further, in step 4, the upper computer performs motion planning on the mechanical arm in the ROS system according to the spatial pose information of the object to be grasped in the robot arm coordinate system and the spatial pose information of the mechanical arm. The specific process of the corresponding motion information queue is: Step 41: The host computer writes a robot arm model description file of the robot arm under the ROS system; Step 42: The host computer reads the robot arm according to the robot arm model description file Performing modeling; Step 43: After modeling the robot arm, the host computer according to the spatial pose information of the object to be grasped in the robot arm coordinate system and the spatial pose information of the robot arm The robot arm performs motion planning to obtain a corresponding motion information queue.

Further, the robot arm grabbing method based on the ROS system further includes: Step 7: while the driving robot arm performs the grabbing operation according to the corresponding path, the lower computer returns the real-time spatial pose information of the robot arm to the The host computer is described; Step 8: The host computer updates the spatial pose information of the robot arm with the real-time spatial pose information returned.

The present disclosure also provides a robotic arm grabbing system based on a ROS system, comprising: a host computer, a lower computer and a camera, wherein the upper computer is communicably connected to the lower computer and the camera; the camera is used for An image of the object to be grasped containing the positioning mark is acquired under the control of the upper computer; the upper computer includes: an image processing module, configured to process the acquired image under the ROS system to obtain the image to be captured Taking spatial pose information of the object in the robot arm coordinate system; the motion planning module is configured to: according to the spatial pose information of the object to be grasped in the robot arm coordinate system and the spatial pose information of the robot arm in the ROS Performing motion planning on the robot arm to obtain a corresponding motion information queue; a message transmission module, configured to transmit the obtained motion information queue of the robot arm to the lower computer; the lower computer includes: motion execution And a module, configured to drive the robot arm according to the motion information queue to perform a grab operation according to a corresponding path.

Further, the host computer further includes: a camera training module, configured to perform positioning mark training on the camera.

Further, the camera training module is configured to perform positioning mark training on the camera, comprising: performing, according to an ARToolKit-based positioning mark recognition algorithm, the camera to perform the positioning mark; or, based on the OpenCV_ArUco-based positioning mark recognition algorithm, The camera is trained for the positioning mark.

Further, the host computer further includes: a camera configuration module, configured to configure the camera node to drive the camera under the ROS system; and, under the ROS system, calibrate the camera and save the correction data; and, select the positioning Tag recognition algorithm.

Further, the image processing module is configured to process the acquired image under the ROS system, and obtain the spatial pose information of the object to be grasped in the robot arm coordinate system, which specifically includes: acquiring at the camera Locating the positioning mark with the highest degree of matching with the preset positioning mark; and positioning the found positioning mark; and, according to the positioned positioning mark, obtaining the object to be grasped with the positioning mark at camera coordinates The spatial pose information of the system; and, according to the preset camera coordinate system and the robot arm coordinate system conversion matrix, converting the spatial pose information of the object to be grasped in the camera coordinate system, and obtaining the The spatial pose information of the object to be grabbed in the robot arm coordinate system.

Further, the motion planning module is configured to perform motion planning on the robot arm according to the spatial pose information of the object to be grasped in a robot arm coordinate system and spatial pose information of the robot arm under the ROS system. Obtaining a corresponding motion information queue specifically includes: writing a robot arm model description file of the robot arm under the ROS system; and modeling the robot arm according to the robot arm model description file; and, when the machine is After the arm is modeled, the upper computer performs motion planning on the mechanical arm according to the spatial pose information of the object to be grasped in the robot arm coordinate system and the spatial pose information of the mechanical arm, and obtains corresponding motion. Information queue.

Further, the lower position machine further includes: an information returning module, configured to return the real-time spatial pose information of the mechanical arm to the upper-level machine while the driving robot arm performs the grasping operation according to the corresponding path; The motion planning module is further configured to update the spatial pose information of the robot arm with the returned real-time spatial pose information.

Compared with the prior art, a ROS system-based mechanical arm grasping method and system of the present invention has a significant advantage in that: firstly, the technical solution of the present disclosure can greatly improve the mechanical arm by introducing machine vision as a core detector component of the mechanical arm. Adaptability, for different items, as long as it is within the working space of the robot arm and the article is within the load range of the robot arm, the grabbing and placing actions can be performed; secondly, the solution adopts a distributed system framework, The upper computer and the lower computer are separated, which can effectively utilize the supercomputer's high computing power and image processing capability, and at the same time To ensure the real-time performance of the lower computer; again, the solution proposed by the present disclosure is based on the ROS operating system, making full use of the rich software package of the ROS system, realizing the rapid configuration of the robot arm motion planning, greatly reducing the threshold of the mechanical arm control; The overall solution proposed by the present disclosure can easily realize the layout of the mechanical arm, and is convenient to expand into a single upper machine with multiple mechanical arms working together, reducing the use cost of the mechanical arm, and has wide application prospects.

DRAWINGS

1 is a schematic diagram of an implementation environment of a robot arm grabbing method based on a ROS system according to the present disclosure;

2 is a working flow chart of a robot arm grabbing method based on a ROS system according to the present disclosure;

3 is a flow chart showing the operation of the upper computer in performing image processing under the ROS system according to the present disclosure;

4 is a working flow chart of the robot arm movement planning performed by the host computer under the ROS system according to the present disclosure;

FIG. 5 is a flow chart of the communication between the upper computer and the lower computer according to the present disclosure;

6 is a schematic structural view of an embodiment of a robot arm grabbing system based on a ROS system according to the present disclosure;

FIG. 7 is a schematic structural view of another embodiment of a robot arm grabbing system based on a ROS system according to the present disclosure.

detailed description

As shown in FIG. 1, according to a preferred embodiment of the present invention, a robot arm grabbing method based on a ROS system, the implementation environment thereof includes a host computer, a lower computer, a camera, and a communication environment. As the main body of image detection and image processing, the upper computer acts as the main body to accept commands and perform crawling, and cooperates to complete the robot arm grabbing task.

Referring to the implementation environment diagram shown in FIG. 1, the implementation environment of this embodiment has the following components:

1) Camera, for example: USB camera. The camera is placed above or obliquely above the object to be grabbed, It is best to have clear and unobstructed shooting angles, and it is necessary to clarify the coordinate system in which the camera is located (coordinate system 1, which is shown in Fig. 1, which is the camera coordinate system).

2) Host computer. The host computer needs to be equipped with the ROS operating system (based on Linux), which is the "brain" of the whole system. Its main functions are: driving the camera to complete image acquisition and transmission, image processing, motion planning, and motion information queue transmission.

3) Lower position machine. The lower position machine refers to the drive control part of the mechanical arm device. Its main function is to receive the motion information queue, the drive robot arm, the real-time spatial pose information sensing detection of the mechanical arm, and the real-time spatial pose information transmission.

4) Robot arm. The robot arm is the execution part of the robot arm device, which requires the robot arm to have more than five degrees of freedom (to ensure that the working space of the robot arm is large, and the grasping operation based on visual positioning can be realized), and the arm has an end effector ( For example: suction cup, end clamp, etc.), different mechanical arm order can be adjusted according to the actual situation when the mechanical arm is modeled. In addition, the position of the robot arm must be determined and modeled based on the position of the arm (coordinate system 2 shown in Figure 1), that is, the robot arm coordinate system and the robot arm in the robot arm coordinate system are required. Spatial pose information.

5) Local area wireless network. Here Socket communication requires a wireless network in the implementation environment, and the upper and lower computers implement communication in the same domain segment.

6) Objects to be grabbed and positioning marks. The requirement for the object to be grasped is that it is placed within the movement space of the robot arm and smaller than the rated load of the robot arm, thereby ensuring that the robot arm can grasp the object. Positioning marks refer to identification patterns with specific shape requirements for visual recognition and positioning. For different algorithms, the identification patterns are different. It should be noted that when placing an object to be grasped containing a positioning mark, the positioning mark needs to be located within the visual range of the camera to ensure that the camera can capture an image of the object to be grasped containing the positioning mark.

In an embodiment of the present disclosure, a robot arm grabbing method based on a ROS system includes:

Step 2: The host computer acquires an image of the object to be grasped including the positioning mark by using a camera;

Specifically, the host computer and the camera are communicatively connected. For example, when the camera is a USB camera, the USB camera can be directly connected to the USB interface of the host computer through the USB interface to implement the connection. Because the host computer is a ROS system, it is necessary to configure the camera's environment under the ROS system to ensure the normal use of the camera.

Therefore, preferably, before step 2, the method further includes: Step 1: The upper computer configures a use environment of the camera under the ROS system.

When the configuration is completed, the upper computer can control the camera to take an image, and the image refers to a picture or an image. In this embodiment, the upper camera controls the camera to capture an image of the object to be grasped with the positioning mark in its visual range.

The image captured by the camera will be transmitted to the host computer, and the image captured by the host computer will be processed to obtain the spatial pose information of the object to be grasped in the robot arm coordinate system. The spatial position information refers to the spatial position and the spatial attitude. For example, if the object to be grasped is a cup, the spatial position refers to the coordinate of the cup in the coordinate system of the robot arm, and the spatial posture means that the cup is placed vertically. Still horizontally. The spatial position is intended to plan the movement of the robot arm to the cup during motion planning. The spatial attitude is to plan whether the end effector of the robot arm reaches the position of the cup after the movement planning, whether it is horizontal or vertical.

The spatial pose information of the object to be grasped in the robot arm coordinate system is transmitted to the MoveIt initialization program module for motion planning. The initial value of the spatial pose information of the robot arm is set in the ROS system from the beginning, and can directly obtain the spatial pose information (equivalent to the end point) and the setting in the robot arm coordinate system according to the processed object to be grasped. Robotic arm in ROS system The spatial pose information (equivalent to the starting point) directly performs motion planning to obtain a motion information queue.

The Socket communication protocol is used for communication between the upper computer and the lower computer, and the upper computer sends the motion information queue to the lower computer through the Socket communication protocol.

After receiving the motion information queue, the lower computer parses the motion information queue, and according to the parsed motion information queue, the robot arm moves according to the corresponding path, and performs a grab operation.

In this embodiment, the image of the object to be grasped by the camera is captured by the camera to locate the spatial pose information of the object to be grasped (this information needs to be in the same coordinate system as the spatial pose information of the robot arm), and then The rules of motion, thereby driving the robotic arm to perform a grasping operation on the grasping object.

Combining machine vision (equivalent to the camera) with the robotic arm is equivalent to adding an intelligent "eye" to the robotic arm, which can greatly increase the environmental sensing capability and intelligent decision-making ability of the mechanical arm, thereby further expanding the application field of the mechanical arm. In addition, the present disclosure is based on the development of the ROS system, and utilizes many characteristics of the ROS system to reduce the difficulty in realizing the motion planning of the robot arm and to lower the application threshold of the mechanical arm.

As shown in FIG. 2, a system operation flowchart of a POS system based visual positioning and a robot arm grasping implementation method is provided. The whole implementation process is divided into two parts: upper computer configuration and lower computer configuration.

The following is a further description of the environment in which the upper computer is configured to operate under the ROS system:

11) The host computer configures the camera node under the ROS system to drive the camera. The camera driver is driven in the ROS system of the host computer. The driver node program used in this embodiment is usb_cam. This node will drive the camera and publish the image captured by the camera on the usb_cam/image_raw topic.

12) The host computer calibrates the camera under the ROS system and saves the correction data. The camera is calibrated using the camera_calibration procedure of the ROS system and the correction data is saved. After driving the camera, the program is used to obtain calibration data of the camera, that is, internal parameters, external parameters, and distortion coefficient data, and the above data is saved as correction data. The correction data obtained by different cameras will be different, and the corrected images will be used to correct the pictures taken by the camera, and then the images with less distortion will be obtained.

13) The host computer selects (trained) the positioning mark recognition algorithm. There may be a variety of positioning mark recognition algorithms inside the camera. When it is necessary to use the camera to control the robot arm for grasping, it is necessary to select a positioning mark recognition algorithm used in the subsequent positioning process for the camera to use for positioning. The positioning mark recognition algorithm may be an ARToolKit-based positioning mark recognition algorithm, an OpenCV_ArUco-based positioning mark recognition algorithm, etc., as long as it can realize visual positioning and let the upper machine control the robot arm to perform the grasping operation.

In another embodiment of the present invention, in addition to the above, the step 1 configuration machine of the camera before the use environment of the ROS system further includes: Step 0: The host computer performs positioning mark training on the camera. Preferably, the process of performing the positioning mark training on the camera by the upper computer in step 0 is any of the following steps:

Step 01: The host computer performs training on the positioning mark for the camera based on an ARToolKit positioning mark recognition algorithm.

Step 02: The host computer trains the camera for the positioning mark based on an OpenCV_ArUco positioning mark recognition algorithm.

Specifically, as long as there is a positioning mark recognition algorithm, the subsequent identification work can be completed. Therefore, only one type of positioning mark recognition algorithm can be trained, and only one algorithm is selected in subsequent selection; if both algorithms are trained You can choose one of the following options.

The identification training methods of the two recognition algorithms proposed by the present disclosure are as follows:

For the ARToolKit-based positioning mark recognition algorithm, you can use the online tool "Tarotaro" training or use the offline tool of mk_patt provided by ARToolKit for training;

For the recognition algorithm based on OpenCV_ArUco, the drawMarker() function is used to create and train the logo pattern (ie, the anchor mark).

In another embodiment of the present disclosure, in addition to the same as the above, the host computer processes the acquired image under the ROS system to obtain the object to be grasped with the positioning mark. The specific process of spatial pose information in the robot arm coordinate system is:

Step 31: Search for an image with the highest matching degree with the preset positioning mark in the image acquired by the camera;

Step 32: Locating the found positioning mark;

Step 33: Obtain spatial pose information of the object to be grasped with the positioning mark in a camera coordinate system according to the positioned positioning mark.

Step 34: Convert the spatial pose information of the object to be grasped in the camera coordinate system according to a preset camera coordinate system and a robot arm coordinate system conversion matrix, and obtain the object to be grasped on the robot arm Spatial pose information in the coordinate system.

Specifically, as shown in FIG. 3, the working flow chart of the image processing of the host computer under the ROS system is completed on one ROS node, and further explained as follows:

Before processing the image acquired by the camera, you first need to read the camera's correction data and select the recognition algorithm to be used (this step is done when configuring the camera's environment in the ROS system).

If the ARToolKit-based positioning mark recognition algorithm is selected, the program processing process is: reading the pre-imported identification map information (ie, reading the preset positioning mark), and the (USB) camera acquires the real-time image to find the most matching positioning identifier and The positioning identifier observed by the camera is positioned, so that the spatial pose information of the object to be grasped with the positioning marker in the camera coordinate system is obtained according to the positioning marker.

It should be noted that the above process refers to the positioning method of the still picture. In actual use, the object to be captured with the positioning mark captured by the camera is a slowly moving image placed on the pipeline. At this time, the ARToolKit frame frequency counter needs to be added, and the position identification map of the position identification of the different frame is estimated. Pose in the camera coordinate system. Here, it is equivalent to dividing the image of the multi-frame into one frame and one frame. In the above manner, the spatial pose information of the object to be grasped in the camera coordinate system of each frame is obtained. When the spatial pose information of the frame to be grabbed in the camera coordinate system is obtained, it will be released in real time for subsequent processing.

In addition, in other embodiments, when debugging is needed, the positioning mark marked with the OpenGL is used as the origin, and the coordinate system is drawn on the icon of the positioning mark, so that the actual display to be captured can be displayed on the monitor display screen. The size, shape, movement, etc. of the object, of course, the movement of the arm will also be displayed on the display, which can more intuitively and dynamically understand the grasping of the arm.

If you choose the OpenCV_ArUco-based positioning mark recognition algorithm, you first need to use ROS. The cv_bridge node is configured in the system to convert the sensor_msgs/Image type image data obtained by the camera under the ROS system into cv::Mat type image data recognizable by the OpenCV library.

Then, in the image acquired by the converted camera, the positioning mark with the highest matching degree with the preset positioning mark is found, and the spatial pose information in the camera coordinate system is obtained. When the acquired image is also an image, the positioning mark with the highest matching degree with the preset positioning mark is sequentially searched from one frame image, and the spatial pose information of the positioned positioning mark in the camera coordinate system is sequentially extracted. The spatial pose information of the object to be grabbed in the camera coordinate system is extracted as a parameter number of a motion rule.

When the positioning mark recognition algorithm of OpenCV_ArUco is selected to identify the positioning mark, the image of the object to be grasped with the positioning mark acquired by the camera (of course, the converted cv::Mat type image data) is used, and then the OTSU binary value is utilized. The algorithm reads the value of the positioning mark in the perspective transformed image and compares it with the value of the pre-trained positioning mark (ie, the value of the preset positioning mark), thereby realizing the identification of the positioning mark.

Then, the spatial position and posture estimation of the identified positioning mark is performed, and the spatial pose information of the object to be grasped in the camera coordinate system is given.

Then, according to the preset camera coordinate system and the robot arm coordinate system conversion matrix, the spatial pose information of the object to be grasped in the camera coordinate system is converted, and the spatial pose of the object to be grasped in the robot arm coordinate system is obtained. information. The spatial pose information of the object to be grabbed in each camera frame in the camera coordinate system needs to be converted.

Finally, the MoveIt interface program is written, and the spatial pose information of the object to be grabbed in the robot arm coordinate system is formatted into a quaternion form and transmitted to the MoveIt initialization program module for motion planning using the API provided by the MoveIt module.

In general, when the upper computer processes the image acquired by the camera, it needs to obtain the spatial pose information of the object to be grasped in each frame of the frame in the robot arm coordinate system, and convert it into a MoveIt initialization program module support. The quaternion form for subsequent motion planning.

FIG. 4 is a flow chart showing the operation of the upper arm machine for manipulating the arm movement under the ROS system according to the present disclosure. Step 4 The upper computer according to the space position of the object to be grasped in the robot arm coordinate system The pose information and the spatial pose information of the manipulator are used to plan the motion of the robot arm under the ROS system, and the corresponding motion information queue is further described as follows:

41. The upper machine (using the URDF (Unified Robot Description Format)) writes a manipulator model description file of the robot arm under the ROS system for modeling the subsequent manipulator.

42. The host computer models the robot arm according to the robot arm model description file. The robotic arm is modeled by calling the created robotic arm description model with the MoveIt Setup Assistant Tool under the ROS system.

Further, the steps of modeling are: collision detection setting, virtual joint setting (for example: the base of the robot arm for positioning of the robot arm coordinate system), and the arm planning joint set (the kinematics solver is KDL Kinematics Plugin) , the initial position of the arm (ie the initial value of the space pose information of the arm), the end of the arm set (for example: define it is a suction cup, clip, etc.), the passive joint setting (specifically, no drive, only The joints that move along with other joints are finally generated. If the motion algorithm is not changed, the default motion algorithm planning library is OMPL (Open Motion Planning Library).

43. After modeling the robot arm, the host computer according to the spatial pose information of the object to be grasped in the robot arm coordinate system and the spatial pose information of the robot arm (using the MoveIt initialization program module) The manipulator performs motion planning, obtains a corresponding motion information queue, and publishes it (according to the communication rules of the ROS system).

FIG. 5 is a flow chart showing the working of the host computer and the lower computer according to the present disclosure. The process is completed on a ROS node, and further explained as follows:

First, the message server program of the ROS system is initialized for reading the motion information queue issued by the MoveIt initialization program module;

Then, the Socket communication node (TCP information) is initialized, and the read motion planning information queue is placed in the sending buffer, and sent to the lower computer when the upper and lower computers communicate;

After that, the lower computer receives the motion planning information, performs motion information analysis, and drives the robot arm to perform the crawling according to the planned action.

In another embodiment of the present disclosure, in addition to the same as the above, the method further includes: Step 7: realizing the robot arm while the driving robot arm performs the grasping operation according to the corresponding path. The spatial pose information is transmitted back to the upper computer; Step 8: The upper computer updates the spatial pose information of the mechanical arm with the real-time spatial pose information returned.

Specifically, as mentioned above, the mechanical arm may need to grasp a plurality of objects to be grasped on the position pipeline, and when the control robot performs a grasping operation, the spatial posture information of the mechanical arm will definitely change. Therefore, at the same time as the planning action is performed, the position sensor (such as the angle sensor and the encoder) of the lower position machine on the robot arm transmits the actual spatial pose information of the robot arm to the upper computer through the Socket communication, so that the upper computer updates the robot arm. The spatial pose information is used for motion planning of the spatial pose information of the next object to be grasped in the robot arm coordinate system.

For the sake of understanding, a specific example is used: a number of objects to be grasped are slowly moved on the pipeline, and the positioning marks of the camera have been trained (using the above two positioning marker recognition algorithms to train). First, the host computer configures the camera's use environment (that is, drives the camera, reads the correction data saved in advance, and the positioning mark recognition algorithm that the engineer can choose to use); then the host computer controls the camera to start shooting on the assembly line. The image of the feature to be captured is released while being shot. When the image processing unit inside the host computer reads the real-time image, it is divided into one frame and one frame to identify the positioning mark. The positioning mark of the frame is released after the spatial pose information of the robot arm coordinate system; the process of processing the image by calling the MoveIt initialization program module and the configuration process of the camera can be executed in parallel, so that it can be like the positioning mark training for the camera Modeling the robotic arm from the beginning, and then monitoring whether to release the spatial pose information of the feature to be captured in the robot arm coordinate system, assuming that there is spatial pose information of the first frame of the feature to be grasped in the robot arm coordinate system, The spatial pose information of the robot arm coordinate system and the spatial position of the robot arm according to the feature to be grasped of the first frame The information (initial value) is used for motion planning, and the corresponding motion information queue is obtained; the lower computer drives the robot arm to perform the grab operation according to the motion information queue, and uploads the real-time spatial pose information of the robot arm to the upper computer; The MoveIt initialization program module updates the spatial pose information of the robot arm according to the uploaded real-time spatial pose information; and then retrieves the spatial pose information of the second frame of the to-be-grabbed feature in the robot arm coordinate system, according to the updated robot arm The spatial pose information is used for motion planning, and the corresponding motion information queue is obtained. The lower computer performs the grab operation according to the motion information queue, and uploads the real-time spatial pose information of the robot arm to the upper computer... thus looping to realize ROS-based systematic Visual positioning and robotic arm grabbing operations.

In summary, the present disclosure has a significant advantage over the prior art in that the technical solution of the present disclosure adopts a distributed design, which is advantageous for utilizing the processing capability of the upper computer and facilitating the topology to cooperate with multiple mechanical arms; The proposed vision-based object localization method is suitable for grasping different objects, and the initial position requirement of the object is low; the mechanical arm motion planning method proposed by the present disclosure fully utilizes the characteristics of the ROS system, and the configuration is simple, convenient and practical; The overall solution uses wireless communication and flexible layout, which can be applied to different application scenarios.

In another embodiment of the present disclosure, as shown in FIG. 6, a robotic arm grabbing system based on a ROS system includes: a host computer 10, a lower computer 30, and a camera 20, wherein the upper computer 10 and the The lower unit 30 and the camera 20 are communicatively coupled.

The upper computer can be a computer with ROS, and the lower computer refers to the drive control part in the mechanical arm device. In this embodiment, the lower computer and the upper position use Socket communication, and the camera needs to communicate with the upper computer, for example: using USB The camera can be connected to the host computer through the USB interface to realize the communication between the two.

The camera 20 is configured to acquire an image of an object to be grasped including a positioning mark under the control of the upper computer;

The host computer 10 further includes:

The image processing module 12 is configured to process the acquired image under the ROS system to obtain spatial pose information of the object to be grasped in a robot arm coordinate system;

The motion planning module 13 is configured to perform motion planning on the mechanical arm under the ROS system according to the spatial pose information of the object to be grasped in the robot arm coordinate system and the spatial pose information of the mechanical arm, and obtain corresponding Motion information queue;

a message passing module 14 is configured to transmit the obtained motion information queue of the robot arm to the lower computer;

The lower machine 30 includes:

The motion execution module 31 is configured to perform a grab operation according to the corresponding path according to the motion information queue driving robot arm.

Specifically, because the host computer is a ROS system, it is necessary to configure the camera under the ROS system. The use environment ensures the normal use of the camera. Preferably, the host computer 10 further includes: a camera configuration module 11 configured to configure a use environment of the camera under the ROS system. When the configuration is completed, the upper computer can control the camera to take an image, and the image refers to a picture or an image. In this embodiment, the upper camera controls the camera to capture an image of the object to be grasped with the positioning mark in its visual range.

The spatial pose information of the object to be grasped in the robot arm coordinate system is transmitted to the MoveIt initialization program module for motion planning. The initial value of the spatial pose information of the robot arm is set in the ROS system from the beginning, and can directly obtain the spatial pose information (equivalent to the end point) and the setting in the robot arm coordinate system according to the processed object to be grasped. The spatial pose information (equivalent to the starting point) of the robot arm in the ROS system directly performs motion planning to obtain a motion information queue.

Combining machine vision (equivalent to the camera) with the robotic arm is equivalent to adding an intelligent "eye" to the robotic arm, which can greatly increase the environmental sensing capability and intelligent decision-making ability of the mechanical arm, thereby further expanding the application field of the mechanical arm. In addition, the present disclosure is based on the development of the ROS system, and utilizes many characteristics of the ROS system to reduce the difficulty of implementing the motion planning of the robot arm and downgrade. The application threshold of the mechanical arm.

Preferably, the camera configuration module 11 is configured to configure the use environment of the camera under the ROS system, specifically: configuring a camera node to drive the camera under the ROS system; and calibrating the camera under the ROS system And save the correction data; and, select (trained) the positioning marker recognition algorithm.

Specifically, the camera driver is performed in the ROS system of the host computer. The driver node program used in this embodiment is usb_cam. This node will drive the camera and publish the image captured by the camera on the usb_cam/image_raw topic.

The camera is calibrated using the camera_calibration procedure of the ROS system and the correction data is saved. After driving the camera, the program is used to obtain calibration data of the camera, that is, internal parameters, external parameters, and distortion coefficient data, and the above data is saved as correction data. The correction data obtained by different cameras will be different, and the corrected images will be used to correct the pictures taken by the camera, and then the images with less distortion will be obtained.

There may be a variety of positioning mark recognition algorithms inside the camera. When it is necessary to use the camera to control the robot arm for grasping, it is necessary to select a positioning mark recognition algorithm used in the subsequent positioning process for the camera to use for positioning. The positioning mark recognition algorithm may be an ARToolKit-based positioning mark recognition algorithm, an OpenCV_ArUco-based positioning mark recognition algorithm, etc., as long as it can realize visual positioning and let the upper machine control the robot arm to perform the grasping operation.

In another embodiment of the present disclosure, in addition to the same as described above, as shown in FIG. 7, the host computer further includes: a camera training module 15 for performing positioning mark training on the camera.

Preferably, the camera training module 15 is configured to perform positioning mark training on the camera, comprising: performing, according to an ARToolKit-based positioning mark recognition algorithm, the camera for the positioning mark; or, based on an OpenCV_ArUco-based positioning mark recognition algorithm, The camera is trained for the positioning mark.

Specifically, in this embodiment, any type of positioning mark recognition algorithm may be used for training to facilitate subsequent recognition. For the training process of the two, please refer to the corresponding method embodiment, which will not be described here.

In another embodiment of the present disclosure, the image processing module is the same as the above 12, for processing the acquired image under the ROS system, and obtaining the spatial pose information of the object to be grasped in the robot arm coordinate system specifically includes:

Searching for an image with the highest matching degree with the preset positioning mark in the image acquired by the camera;

And positioning the located positioning mark;

And obtaining, according to the positioning marker, the spatial pose information of the object to be grasped with the positioning marker in a camera coordinate system;

And converting, according to the preset camera coordinate system and the robot arm coordinate system conversion matrix, the spatial pose information of the object to be grasped in the camera coordinate system, and obtaining the coordinates of the object to be grasped at the arm The spatial pose information under the system.

Specifically, if the ARToolKit-based positioning mark recognition algorithm is selected, the program processing process is: reading the pre-imported identification map information (ie, reading the preset positioning mark), and the (USB) camera acquiring the real-time image to find the highest matching degree. Positioning the identifier and locating the positioning identifier observed by the camera, thereby obtaining the spatial pose information of the object to be grasped with the positioning marker in the camera coordinate system according to the positioning marker.

In another embodiment of the present disclosure, in addition to the same as the above, the motion planning module 13 is configured to use the spatial pose information and the machine according to the object to be grasped in the robot arm coordinate system. The spatial pose information of the arm is used to plan the motion of the robot arm under the ROS system, and the corresponding motion information queue includes:

(Us URDF) Write a description of the robotic arm model of the robotic arm under the ROS system.

And modeling the robot arm according to the robot arm model description file. The manipulator description model created by the MoveIt initialization toolkit of the ROS system is used to model the robot arm. For the specific steps of the modeling, please refer to the corresponding method embodiment, which is not described here.

And, after modeling the mechanical arm, the host computer according to the spatial pose information of the object to be grasped in the robot arm coordinate system and the spatial pose information of the mechanical arm (using the MoveIt initialization program module) The motion planning of the robot arm is performed, and the corresponding motion information queue is obtained and released (according to the communication rules of the ROS system).

In this embodiment, the Socket communication is used to enable the upper computer and the lower computer to communicate. For the specific communication process, refer to the corresponding method embodiment, which is not described herein.

In another embodiment of the present disclosure, in addition to the same as the above, as shown in FIG. 7, the lower computer further includes:

The information returning module 32 is configured to return the real-time spatial pose information of the mechanical arm to the upper computer while the driving robot arm performs the grasping operation according to the corresponding path;

The motion planning module is further configured to update the spatial pose information of the robot arm with the returned real-time spatial pose information.

While the invention has been described above in the preferred embodiments, it is not intended to limit the invention. A person skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention is defined by the claims The standard is subject to change.

Claims

A robot arm grabbing method based on a ROS system, comprising:

Step 2: The upper computer obtains an image of the object to be grasped containing the positioning mark through the camera;

Step 3: The upper computer processes the acquired image under the ROS system, and obtains spatial pose information of the object to be grasped in a robot arm coordinate system;

Step 4: The upper computer performs motion planning on the mechanical arm under the ROS system according to the spatial pose information of the object to be grasped in the robot arm coordinate system and the spatial pose information of the mechanical arm, and obtains corresponding Motion information queue;

Step 5: The host computer transmits the obtained motion information queue of the robot arm to the lower position machine;

Step 6: The lower computer drives the robot arm according to the motion information queue to perform a grab operation according to a corresponding path.
The ROS system-based robotic arm grabbing method according to claim 1, wherein the step 1 configuration machine of the camera before the use environment of the ROS system further comprises:

Step 0: The host computer performs positioning mark training on the camera.
The ROS system-based robotic arm grabbing method according to claim 2, wherein the step of performing the positioning mark training on the camera by the host computer comprises any one of the following steps:

Step 01: The host computer performs training on the positioning mark for the camera based on an ARToolKit positioning mark recognition algorithm.

Step 02: The host computer trains the camera for the positioning mark based on an OpenCV_ArUco positioning mark recognition algorithm.
The ROS system-based mechanical arm grasping method according to claim 1, wherein the step 2: the upper computer acquires the object to be grasped including the positioning mark by using a camera The image of the body also includes:

Step 11: The host computer configures a camera node to drive the camera under the ROS system;

Step 12: The host computer calibrates the camera under the ROS system and saves the correction data;

Step 13: The upper computer selects a positioning mark recognition algorithm.
The method for grasping a robot arm based on a ROS system according to claim 1, wherein the upper computer performs processing on the acquired image under the ROS system to obtain the image with the positioning mark. The specific process of grasping the spatial pose information of the object in the robot arm coordinate system is:

Step 31: Search for an image with the highest matching degree with the preset positioning mark in the image acquired by the camera;

Step 32: Locating the found positioning mark;

Step 33: Obtain spatial pose information of the object to be grasped with the positioning mark in a camera coordinate system according to the positioned positioning mark.

Step 34: Convert the spatial pose information of the object to be grasped in the camera coordinate system according to a preset camera coordinate system and a robot arm coordinate system conversion matrix, and obtain the object to be grasped on the robot arm Spatial pose information in the coordinate system.
The ROS system-based robot arm grasping method according to claim 1, wherein the upper computer performs the spatial pose information according to the object to be grasped in a robot arm coordinate system according to the step 4 The spatial pose information of the manipulator is used to plan the motion of the manipulator under the ROS system. The specific process of obtaining the corresponding motion information queue is:

Step 41: The upper computer writes a robot arm model description file of the robot arm under the ROS system;

Step 42: The host computer models the robot arm according to the robot arm model description file;

Step 43: After modeling the mechanical arm, the upper computer according to the object to be grasped The spatial pose information and the spatial pose information of the robot arm in the robot arm coordinate system perform motion planning on the robot arm to obtain a corresponding motion information queue.
The method for grasping a robot arm based on a ROS system according to claim 1, further comprising:

Step 7: while the driving robot arm performs the grasping operation according to the corresponding path, the lower computer returns the real-time spatial pose information of the robot arm to the upper computer;

Step 8: The host computer updates the spatial pose information of the robot arm with the real-time spatial pose information returned.
A robotic arm grabbing system based on a ROS system, comprising: a host computer, a lower computer, and a camera, wherein the upper computer is communicably connected to the lower computer and the camera;

The camera is configured to acquire an image of an object to be grasped including a positioning mark under the control of the upper computer;

The upper computer includes:

An image processing module, configured to process the acquired image under the ROS system, and obtain spatial pose information of the object to be grasped in a robot arm coordinate system;

a motion planning module, configured to perform motion planning on the mechanical arm under the ROS system according to the spatial pose information of the object to be grasped in a robot arm coordinate system and spatial pose information of the mechanical arm, and obtain corresponding Motion information queue;

a message passing module, configured to transmit the obtained motion information queue of the robot arm to the lower position machine;

The lower position machine includes:

And a motion execution module, configured to perform a grab operation according to the corresponding path according to the motion information queue driving robot arm.
The ROS system-based mechanical arm grasping system according to claim 8, The problem is that the upper computer further includes:

A camera training module is configured to perform positioning mark training on the camera.
The ROS system-based robotic arm grabbing system according to claim 9, wherein the camera training module is configured to perform positioning mark training on the camera, including:

The camera is trained for the positioning mark based on an ARToolKit-based positioning mark recognition algorithm; or, according to an OpenCV_ArUco-based positioning mark recognition algorithm, the camera is trained for the positioning mark.
The ROS system-based mechanical arm grasping system according to claim 8, wherein the upper computer further comprises:

a camera configuration module for configuring a camera node to drive the camera under the ROS system; and, calibrating the camera and storing the correction data under the ROS system; and selecting a positioning mark recognition algorithm.
The ROS system-based robotic arm grabbing system according to claim 8, wherein the image processing module is configured to process the acquired image under the ROS system to obtain the object to be grasped. The spatial pose information under the robot arm coordinate system specifically includes:

Searching for an image with the highest matching degree with the preset positioning mark in the image acquired by the camera;

And positioning the located positioning mark;

And obtaining, according to the positioning marker, the spatial pose information of the object to be grasped with the positioning marker in a camera coordinate system;

And converting, according to the preset camera coordinate system and the robot arm coordinate system conversion matrix, the spatial pose information of the object to be grasped in the camera coordinate system, and obtaining the coordinates of the object to be grasped at the arm The spatial pose information under the system.
The ROS system-based mechanical arm grasping system according to claim 8, The motion planning module is configured to move the mechanical arm under the ROS system according to the spatial pose information of the object to be grasped in a robot arm coordinate system and the spatial pose information of the mechanical arm. Planning to get the corresponding sports information queue specifically includes:

Write a robotic arm model description file of the robot arm under the ROS system;

And modeling the robot arm according to the robot arm model description file;

And, after modeling the mechanical arm, the upper computer performs the mechanical arm according to the spatial pose information of the object to be grasped in a robot arm coordinate system and the spatial pose information of the mechanical arm. Motion planning, get the corresponding motion information queue.
The ROS system-based mechanical arm grasping system according to claim 8, wherein the lower position machine further comprises:

The information returning module is configured to return the real-time spatial pose information of the mechanical arm to the upper computer while the driving robot arm performs the grasping operation according to the corresponding path;

The motion planning module is further configured to update the spatial pose information of the robot arm with the returned real-time spatial pose information.