CA2369845A1 - Method and apparatus for single camera 3d vision guided robotics - Google Patents
Method and apparatus for single camera 3d vision guided robotics Download PDFInfo
- Publication number
- CA2369845A1 CA2369845A1 CA002369845A CA2369845A CA2369845A1 CA 2369845 A1 CA2369845 A1 CA 2369845A1 CA 002369845 A CA002369845 A CA 002369845A CA 2369845 A CA2369845 A CA 2369845A CA 2369845 A1 CA2369845 A1 CA 2369845A1
- Authority
- CA
- Canada
- Prior art keywords
- camera
- space
- robot
- frame
- transformation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Classifications
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B25—HAND TOOLS; PORTABLE POWER-DRIVEN TOOLS; MANIPULATORS
- B25J—MANIPULATORS; CHAMBERS PROVIDED WITH MANIPULATION DEVICES
- B25J9/00—Programme-controlled manipulators
- B25J9/16—Programme controls
- B25J9/1694—Programme controls characterised by use of sensors other than normal servo-feedback from position, speed or acceleration sensors, perception control, multi-sensor controlled systems, sensor fusion
- B25J9/1697—Vision controlled systems
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S5/00—Position-fixing by co-ordinating two or more direction or position line determinations; Position-fixing by co-ordinating two or more distance determinations
- G01S5/16—Position-fixing by co-ordinating two or more direction or position line determinations; Position-fixing by co-ordinating two or more distance determinations using electromagnetic waves other than radio waves
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T1/00—General purpose image data processing
- G06T1/0007—Image acquisition
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/10—Image acquisition
- G06V10/12—Details of acquisition arrangements; Constructional details thereof
- G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
- G06V10/147—Details of sensors, e.g. sensor lenses
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
- G05B2219/00—Program-control systems
- G05B2219/30—Nc systems
- G05B2219/39—Robotics, robotics to robotics hand
- G05B2219/39057—Hand eye calibration, eye, camera on hand, end effector
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
- G05B2219/00—Program-control systems
- G05B2219/30—Nc systems
- G05B2219/39—Robotics, robotics to robotics hand
- G05B2219/39393—Camera detects projected image, compare with reference image, position end effector
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30108—Industrial image inspection
- G06T2207/30164—Workpiece; Machine component
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Multimedia (AREA)
- Vascular Medicine (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Mechanical Engineering (AREA)
- Electromagnetism (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Robotics (AREA)
- Manipulator (AREA)
- Image Processing (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
A method of three-dimensional handling of an object by a robot using a tool and one camera mounted on the robot is disclosed in which at least six target features which are normal features of the object are selected on the object.
Description
VISION GUIDED ROBOTICS
Technical Field The invention relates to the field of vision guided robotics, and more particularly to a method and apparatus for single camera three dimensional vision guided robotics.
Back rg ound Robots have long been widely used in manufacturing processes for many applications. Many different types of sensors are used to guide the robot but machine vision is increasingly being used to guide robots in their tasks.
Typically such machine vision is used in a two-dimensional application wherein the target object need only be located in an x-y plane, using a single camera. For example see United States patent no. 4,437,114 LaRussa. However many robotic applications require the robot to locate and manipulate the target in three dimensions. In the past this has involved using two or more cameras. For example see United States patent no. 4,146,924 Birk et al.; and United States patent no. 5,959,425 Bieman et al.
Target points have also been used to assist in determining location in space of the target object using single or multiple cameras. See United States patents no.
5,696,673; 5,956,417; 6,044,183 and 6,301,763 all of Pryor. It is important that computations be carried out rapidly, with minimum computing power requirements.
There is therefore a need for a process for single camera three dimensional vision guided robotics in which computations can be carried out rapidly and with minimum computational power requirements.
Summary of Invention A method for three-dimensional handling of an object by a robot using a tool and one camera mounted on the robot is disclosed in which targets are used on the object. The process is performed in three main steps:
a) calibration of the camera;
b) teaching the targets on the object;
c) finding the position of the object and using the robot for han-dung.
Technical Field The invention relates to the field of vision guided robotics, and more particularly to a method and apparatus for single camera three dimensional vision guided robotics.
Back rg ound Robots have long been widely used in manufacturing processes for many applications. Many different types of sensors are used to guide the robot but machine vision is increasingly being used to guide robots in their tasks.
Typically such machine vision is used in a two-dimensional application wherein the target object need only be located in an x-y plane, using a single camera. For example see United States patent no. 4,437,114 LaRussa. However many robotic applications require the robot to locate and manipulate the target in three dimensions. In the past this has involved using two or more cameras. For example see United States patent no. 4,146,924 Birk et al.; and United States patent no. 5,959,425 Bieman et al.
Target points have also been used to assist in determining location in space of the target object using single or multiple cameras. See United States patents no.
5,696,673; 5,956,417; 6,044,183 and 6,301,763 all of Pryor. It is important that computations be carried out rapidly, with minimum computing power requirements.
There is therefore a need for a process for single camera three dimensional vision guided robotics in which computations can be carried out rapidly and with minimum computational power requirements.
Summary of Invention A method for three-dimensional handling of an object by a robot using a tool and one camera mounted on the robot is disclosed in which targets are used on the object. The process is performed in three main steps:
a) calibration of the camera;
b) teaching the targets on the object;
c) finding the position of the object and using the robot for han-dung.
-2-According to one aspect of the invention, the invention provides a method of three-dimensional handling of an object by a robot using a tool and one camera mounted on the robot, comprising:
i) calibrating the camera by finding a) the camera intrinsic parameter and b) the position of the camera relative to the tool of the robot ( "hand - eye " calibration ) and c) the position of the camera in a space rigid to the place where the object will be trained;
ii) teaching the object features by a) putting the object in the "User Frame " space and capturing an image with the robot in the calibration position where the "Camera - >
User Frame " transformation was calculated;
b) selecting at least one anchor feature from the image and at least 5 other visible features;
c) calculating the 3D position of each feature in "User Frame" using the height of the feature relative to the object base plane and the "Camera - > User Frarne" transformation;
d) defining an "Object Space " aligned with the "User Frame " but connected to the object and transposing the 3D coordinates of the features into that space;
e) computing the "Object Space - > Camera " transforma-tion using the 3D position of the features inside this space and the position in the image;
f) defining an "Object Frame" inside "Object Space " to be used for teaching the handling path;
g) computing the Object Frame position and orientation in "Tool Frame" using the transformation from "Object Frame-> Camera " and "Cam-era - > Tool";
h) sending the "Object Frame " to the robot and training t-he handling path inside the "Object Frame";
iii) carrying out object finding and. positioning by a) positioning the robot in a predefined position above the bin containing the target object;
b) if no object is in the field of view, moving the robot until an anchor feature is found in the image;
i) calibrating the camera by finding a) the camera intrinsic parameter and b) the position of the camera relative to the tool of the robot ( "hand - eye " calibration ) and c) the position of the camera in a space rigid to the place where the object will be trained;
ii) teaching the object features by a) putting the object in the "User Frame " space and capturing an image with the robot in the calibration position where the "Camera - >
User Frame " transformation was calculated;
b) selecting at least one anchor feature from the image and at least 5 other visible features;
c) calculating the 3D position of each feature in "User Frame" using the height of the feature relative to the object base plane and the "Camera - > User Frarne" transformation;
d) defining an "Object Space " aligned with the "User Frame " but connected to the object and transposing the 3D coordinates of the features into that space;
e) computing the "Object Space - > Camera " transforma-tion using the 3D position of the features inside this space and the position in the image;
f) defining an "Object Frame" inside "Object Space " to be used for teaching the handling path;
g) computing the Object Frame position and orientation in "Tool Frame" using the transformation from "Object Frame-> Camera " and "Cam-era - > Tool";
h) sending the "Object Frame " to the robot and training t-he handling path inside the "Object Frame";
iii) carrying out object finding and. positioning by a) positioning the robot in a predefined position above the bin containing the target object;
b) if no object is in the field of view, moving the robot until an anchor feature is found in the image;
-3-c) using the position and orientation of the anchor feature to compute the position of the remaining set of selected features;
d) with the positions of features from the image and their correspondent position in "Object Space"as calculated in the training session, using the camera calibration to compute the transformation between the "Object Space "
and "Camera Space ";
e) using the said transformation to calculate the movement of the robot to position the camera so that it "looks" orthogonal to the object;
fj fording the "Object Space - > Camera Space" transfor-oration in the same way as in step d);
g) computing the object frame memorized at training using the found transformation and "Camera -> Tool " transformation;
h) sending the commuted "Object Frame" to the robot;
i) using the "Tool" position to define the frame in "Robot Space" and performing the trained handling on the object inside the "Robot Space".
Brief Description of Drawings In drawings which illustrate a preferred embodiment of the invention:
Fig. 1 is a perspective view of a vision-guided robot;
Fig. 2 is a flow chart illustrating the calibration of the camera mount-ed on the robot arm;
Fig. 3 is a flow chart illustrating the method of teaching the object features and handling path; and Fig. 4 is a flow chart illustrating the method of object positioning and handling.
Description Throughout the following description, specific details are set forth in order to provide a more thorough understanding of the invention. However, the invention may be practiced without these particulars. In other instances, well known elements have not been shown or described in detail to avoid unnecessarily obscuring the invention. Accordingly, the specification and drawings are to be regarded in an illustrative, rather than a restrictive, sense.
d) with the positions of features from the image and their correspondent position in "Object Space"as calculated in the training session, using the camera calibration to compute the transformation between the "Object Space "
and "Camera Space ";
e) using the said transformation to calculate the movement of the robot to position the camera so that it "looks" orthogonal to the object;
fj fording the "Object Space - > Camera Space" transfor-oration in the same way as in step d);
g) computing the object frame memorized at training using the found transformation and "Camera -> Tool " transformation;
h) sending the commuted "Object Frame" to the robot;
i) using the "Tool" position to define the frame in "Robot Space" and performing the trained handling on the object inside the "Robot Space".
Brief Description of Drawings In drawings which illustrate a preferred embodiment of the invention:
Fig. 1 is a perspective view of a vision-guided robot;
Fig. 2 is a flow chart illustrating the calibration of the camera mount-ed on the robot arm;
Fig. 3 is a flow chart illustrating the method of teaching the object features and handling path; and Fig. 4 is a flow chart illustrating the method of object positioning and handling.
Description Throughout the following description, specific details are set forth in order to provide a more thorough understanding of the invention. However, the invention may be practiced without these particulars. In other instances, well known elements have not been shown or described in detail to avoid unnecessarily obscuring the invention. Accordingly, the specification and drawings are to be regarded in an illustrative, rather than a restrictive, sense.
-4-Fig. 1 shows a robot 10 having a manipulating arm 12 and attached tool 24 with camera 16 mounted thereon. Tool 14 is designed to manipulate a part or target object 18. Camera 16 is preferably a commercially available digital video camera.
The method is performed in three main steps described as follows:
a) calibration of the camera;
b) teaching the targets on the object; and c) finding the position of the object and use the robot for han-dung;
Calibration In the following discussion the following terms have the following meanings:
Transformation: three-dimensional rotation & translation between two spaces;
Tool: the tool the robot is using for performing the handling;
Camera Space: a space rigid to the camera ;
Camera calibration intrinsic parameters: focal length, image center, real pixel size, radial distortion for the lens;
Camera calibration extrinsic parameters: rotation & translation of an external space relative to the camera space.
The calibration process involves i) finding the camera intrinsic parameter and ii) the position of the camera relative to the tool of the robot ( "hand - eye " calibration ). The position of the camera in the " user space ", which is a space rigid to the place where the object will be trained is also determined.
An explanation of the basic calibration algorithms and descriptions of the variables can be found in the following publications:
"An Efficient and Accurate Camera Calibration Technique for 3D Machine Vision", Roger Y. Tsai, Proceedings of IEEE
Conference on Computer Vision and Pattern Recognition, Miami Beach, FL, 1986, pages 364-374;
"A Versatile Camera Calibration Technique for High-Accuracy 3D Machine Vision Metrology Using Off the-Shelf TV Cam-- -eras and Lenses", Roger Y. Tsai, IEEE Journal of Robotics and Automation, Vol. RA-3, No. 4, August 1987, pages 323-344.
"Tsar s Camera Calibration Method Revisited"
http: //www. ai.mit. edu/people/bkph/papers/tsaiexplain. pdf Tsars camera model is based on the pin hole model of perspective projection. Given the position of a point in 3D world coordinates the model predicts the position of the point's image in 2D pixel coordinates. Tsars model has 11 parameters: five internal (also called intrinsic or interior) parameters:
i) f - effective focal length of the pin hole camera;
ii) kappal - 1st order radial lens distortion coefficient;
iii and iv) Cx, Cy - coordinates of center of radial lens distortion and the piercing point of the camera coordinate frame's Z axis with the camera's sensor plane;
v) sx - scale factor to account for any uncertainty due to framegrabber horizontal scanline resampling, and six external (also called extrinsic or exterior) parameters:
i) Rx, Ry, Rz - rotation angles for the transform between the world and camera coordinate frames, and ii) Tx, Ty, Tz - translational components for the transform between the world and camera coordinate frames.
The internal parameters describe how the camera forms an image while the external parameters describe the camera's pose (i.e. position and orienta-tion) in the world coordinate frame. Calibration data for the model consists of 3D
(x,y,z) world coordinates of a feature point (in mm for example) and corresponding 2D coordinates (Xf,Yf) (typically in pixels) of the feature point in the image. Two forms of calibration are possible:
coplanar - the calibration points lie in a single plane in 3D, and non-coplanar - the calibration points occupy a 3D volume.
As illustrated in Fig. 2, the first step in calibration is to position the camera on the robot arm so it is orthogonal to the template ( using the mirror approa-ch), and defining the "User Frame" for the robot aligned with the template used for calibration. Next the camera intrinsic parameter and the "Camera - > User Frame "
transformation are computed considering the user frame. Next, the "Camera - >
Tool" transformation is computed using the "Camera -> User Frame " transforma-tion and inquiring the robot about the "Tool "position in "User Frame" .
Teaching Teaching the object is the process of a) selection from the object's image of a set of at least 6 features, and memorizing the "description" of the features and the relative positions in the image. Features can be edges, holes, corners, blobs ( extracted from the image ) or simply a region of the image witch will be used in a pattern match. First, a large unique feature is selected to be considered anchor feature. The other selected features are small features relative to the first one;
b) Real world coordinates are computed for the selected features.
The object is located in the user space so by using the features' height relative to the bottom of the object, the 3D position of the object features inside the user space can be computed using the position in the image and the "user space" to camera transfor-mation calculated at calibration. All these coordinates are rearranged in a space con-nected to the object ( the first feature is the origin ) and aligned with the "user space" .
c) Also an object frame will be defined as rigid to the object, in a position selected by the user.
d) The object frame (computed in tool coordinates) is sent to the robot to be considered as the working space. To find this frame position, the transformation from object space to camera is used, then from camera to tool.
e) Relative to that space the operator can train the handling path for the robot.
f) Steps c) and d) above can be eliminated and line teaching of the path used instead, since the object frame can be positioned anywhere relative to the object. The CAD model of the object can than be used to define the handling path.
As illustrated in Fig. 3, the first step in teaching the object features is to put the object in the "User Frame " space and capture an image with the robot in the calibration position ( where the "Camera - > User Frame " transformation was calculated ). Next, at least one anchor feature is selected from the image and a set of at least 5 other visible features is selected. The 3D position of each feature is then calculated in "User Frame " using the height of the feature relative to the object base plane and " Camera -> User Frame" transformation. An "Object Space " is then defined aligned with the "User Frame " but connected to the object and the 3D
coordinates of the features are transposed into that space. The "Object Space ->
Camera " transformation is computed using the 3D position of the features inside this space and the position in the image by computing an extrinsic calibration using the camera calibration from the previous step using the Tsai method referenced above.
Next an "Object Frame" inside "Object Space " is defined to be used for teaching the handling path. This Object Frame position and orientation in "Tool Frame" is com-puted using the transformation from "Object Frame- > Camera " and "Camera - >
Tool". The "Object Frame " is then sent to the robot and the handling path is trained inside this space.
1. Object finding & positioning To carry out object fording and positioning, a) the robot is positioned in any predefined position above the bin with objects. b) An image of the object is snapped. c) The trained features are searched using a first search for the anchor feature. Using the position and orientation of the anchor the rest of the relative features can be found. This approach allows similar features to be selected as features are searched in a relatively small region of interest. d) The position (in the image and in the object space) of the found features (at least 6) are used to calculate the transformation between the object space and the camera space using an extrinsic calibration algorithm. ( see the Tsai article above). The found position is used to reorient the camera to "look" to the object from a position close to the one used at training. This last step is necessary mostly if the object has major rotations, so the features are distorted and the found position is not completely accurate.
Steps repeat c) and d) above are repeated. e) The previous space transformation is used in conjunction with the "camera to tool " transformation to find the position of the object frame in tool space. f) The object frame is then sent to the robot to be used as the space for performing the handling;
With reference to Fig. 4, the steps to carry out object finding and positioning are illustrated. First the robot is positioned in a predefined position above the bin with the target object. If no object is in the field of view, the robot is moved until an anchor feature is found in the image. Next the position and orienta-tion of the anchor feature is used to compute the position of the rest of the features.
The position of all the visible features in the image is found. With the positions of features from the image and their correspondent position in "Object Space"
calculated in the training session) the camera calibration is used to compute the transformation between the "Object Space " and "Camera Space ". The camera extrinsic calibration is used as described in the Tsai article referenced above.
Next the transformation described above is used to calculate the movement of the robot to position the camera so that it "looks" orthogonal to the object, namely the same position as in training. In this way all the features will be as similar as possible to the ones at training. This will make the recognition and positioning more accurate. Next the "Object Space -> Camera Space" transforma-tion is found in the same way as in the previous step (using the features positions).
The object frame memorized at training is computed using the found transformation and "Camera - > Tool " transformation. Next, the commuted "Object Frame" is sent to the robot. The "Tool" position is used to define the frame in "Robot Space"
. The trained handling is performed on the object inside this space.
Thus methods for teaching robots and handling of objects by robots in three dimensions using one camera mounted on the robot arm are disclosed in which targets are used on objects. The targets are normal features of the object. It is not necessary to use the CAD design of the object according to this method. The objects are not fixed in fixtures and axe not searched for in a. specific place. While the method has been described for one trained object, the process can be used in the same manner to first recognize the object and then find its location in three dimen-sions. Also the method has been described for one visible set of features selected on one side of the object, but it can be extended to all the sides that can appear in a working situation.
As will be apparent to those skilled in the art in the light of the foregoing disclosure, many alterations and modifications are possible in the practice of this invention without departing from the spirit or scope thereof.
Accordingly, the scope of the invention is to be construed in accordance with the substance defined by the following claims.
The method is performed in three main steps described as follows:
a) calibration of the camera;
b) teaching the targets on the object; and c) finding the position of the object and use the robot for han-dung;
Calibration In the following discussion the following terms have the following meanings:
Transformation: three-dimensional rotation & translation between two spaces;
Tool: the tool the robot is using for performing the handling;
Camera Space: a space rigid to the camera ;
Camera calibration intrinsic parameters: focal length, image center, real pixel size, radial distortion for the lens;
Camera calibration extrinsic parameters: rotation & translation of an external space relative to the camera space.
The calibration process involves i) finding the camera intrinsic parameter and ii) the position of the camera relative to the tool of the robot ( "hand - eye " calibration ). The position of the camera in the " user space ", which is a space rigid to the place where the object will be trained is also determined.
An explanation of the basic calibration algorithms and descriptions of the variables can be found in the following publications:
"An Efficient and Accurate Camera Calibration Technique for 3D Machine Vision", Roger Y. Tsai, Proceedings of IEEE
Conference on Computer Vision and Pattern Recognition, Miami Beach, FL, 1986, pages 364-374;
"A Versatile Camera Calibration Technique for High-Accuracy 3D Machine Vision Metrology Using Off the-Shelf TV Cam-- -eras and Lenses", Roger Y. Tsai, IEEE Journal of Robotics and Automation, Vol. RA-3, No. 4, August 1987, pages 323-344.
"Tsar s Camera Calibration Method Revisited"
http: //www. ai.mit. edu/people/bkph/papers/tsaiexplain. pdf Tsars camera model is based on the pin hole model of perspective projection. Given the position of a point in 3D world coordinates the model predicts the position of the point's image in 2D pixel coordinates. Tsars model has 11 parameters: five internal (also called intrinsic or interior) parameters:
i) f - effective focal length of the pin hole camera;
ii) kappal - 1st order radial lens distortion coefficient;
iii and iv) Cx, Cy - coordinates of center of radial lens distortion and the piercing point of the camera coordinate frame's Z axis with the camera's sensor plane;
v) sx - scale factor to account for any uncertainty due to framegrabber horizontal scanline resampling, and six external (also called extrinsic or exterior) parameters:
i) Rx, Ry, Rz - rotation angles for the transform between the world and camera coordinate frames, and ii) Tx, Ty, Tz - translational components for the transform between the world and camera coordinate frames.
The internal parameters describe how the camera forms an image while the external parameters describe the camera's pose (i.e. position and orienta-tion) in the world coordinate frame. Calibration data for the model consists of 3D
(x,y,z) world coordinates of a feature point (in mm for example) and corresponding 2D coordinates (Xf,Yf) (typically in pixels) of the feature point in the image. Two forms of calibration are possible:
coplanar - the calibration points lie in a single plane in 3D, and non-coplanar - the calibration points occupy a 3D volume.
As illustrated in Fig. 2, the first step in calibration is to position the camera on the robot arm so it is orthogonal to the template ( using the mirror approa-ch), and defining the "User Frame" for the robot aligned with the template used for calibration. Next the camera intrinsic parameter and the "Camera - > User Frame "
transformation are computed considering the user frame. Next, the "Camera - >
Tool" transformation is computed using the "Camera -> User Frame " transforma-tion and inquiring the robot about the "Tool "position in "User Frame" .
Teaching Teaching the object is the process of a) selection from the object's image of a set of at least 6 features, and memorizing the "description" of the features and the relative positions in the image. Features can be edges, holes, corners, blobs ( extracted from the image ) or simply a region of the image witch will be used in a pattern match. First, a large unique feature is selected to be considered anchor feature. The other selected features are small features relative to the first one;
b) Real world coordinates are computed for the selected features.
The object is located in the user space so by using the features' height relative to the bottom of the object, the 3D position of the object features inside the user space can be computed using the position in the image and the "user space" to camera transfor-mation calculated at calibration. All these coordinates are rearranged in a space con-nected to the object ( the first feature is the origin ) and aligned with the "user space" .
c) Also an object frame will be defined as rigid to the object, in a position selected by the user.
d) The object frame (computed in tool coordinates) is sent to the robot to be considered as the working space. To find this frame position, the transformation from object space to camera is used, then from camera to tool.
e) Relative to that space the operator can train the handling path for the robot.
f) Steps c) and d) above can be eliminated and line teaching of the path used instead, since the object frame can be positioned anywhere relative to the object. The CAD model of the object can than be used to define the handling path.
As illustrated in Fig. 3, the first step in teaching the object features is to put the object in the "User Frame " space and capture an image with the robot in the calibration position ( where the "Camera - > User Frame " transformation was calculated ). Next, at least one anchor feature is selected from the image and a set of at least 5 other visible features is selected. The 3D position of each feature is then calculated in "User Frame " using the height of the feature relative to the object base plane and " Camera -> User Frame" transformation. An "Object Space " is then defined aligned with the "User Frame " but connected to the object and the 3D
coordinates of the features are transposed into that space. The "Object Space ->
Camera " transformation is computed using the 3D position of the features inside this space and the position in the image by computing an extrinsic calibration using the camera calibration from the previous step using the Tsai method referenced above.
Next an "Object Frame" inside "Object Space " is defined to be used for teaching the handling path. This Object Frame position and orientation in "Tool Frame" is com-puted using the transformation from "Object Frame- > Camera " and "Camera - >
Tool". The "Object Frame " is then sent to the robot and the handling path is trained inside this space.
1. Object finding & positioning To carry out object fording and positioning, a) the robot is positioned in any predefined position above the bin with objects. b) An image of the object is snapped. c) The trained features are searched using a first search for the anchor feature. Using the position and orientation of the anchor the rest of the relative features can be found. This approach allows similar features to be selected as features are searched in a relatively small region of interest. d) The position (in the image and in the object space) of the found features (at least 6) are used to calculate the transformation between the object space and the camera space using an extrinsic calibration algorithm. ( see the Tsai article above). The found position is used to reorient the camera to "look" to the object from a position close to the one used at training. This last step is necessary mostly if the object has major rotations, so the features are distorted and the found position is not completely accurate.
Steps repeat c) and d) above are repeated. e) The previous space transformation is used in conjunction with the "camera to tool " transformation to find the position of the object frame in tool space. f) The object frame is then sent to the robot to be used as the space for performing the handling;
With reference to Fig. 4, the steps to carry out object finding and positioning are illustrated. First the robot is positioned in a predefined position above the bin with the target object. If no object is in the field of view, the robot is moved until an anchor feature is found in the image. Next the position and orienta-tion of the anchor feature is used to compute the position of the rest of the features.
The position of all the visible features in the image is found. With the positions of features from the image and their correspondent position in "Object Space"
calculated in the training session) the camera calibration is used to compute the transformation between the "Object Space " and "Camera Space ". The camera extrinsic calibration is used as described in the Tsai article referenced above.
Next the transformation described above is used to calculate the movement of the robot to position the camera so that it "looks" orthogonal to the object, namely the same position as in training. In this way all the features will be as similar as possible to the ones at training. This will make the recognition and positioning more accurate. Next the "Object Space -> Camera Space" transforma-tion is found in the same way as in the previous step (using the features positions).
The object frame memorized at training is computed using the found transformation and "Camera - > Tool " transformation. Next, the commuted "Object Frame" is sent to the robot. The "Tool" position is used to define the frame in "Robot Space"
. The trained handling is performed on the object inside this space.
Thus methods for teaching robots and handling of objects by robots in three dimensions using one camera mounted on the robot arm are disclosed in which targets are used on objects. The targets are normal features of the object. It is not necessary to use the CAD design of the object according to this method. The objects are not fixed in fixtures and axe not searched for in a. specific place. While the method has been described for one trained object, the process can be used in the same manner to first recognize the object and then find its location in three dimen-sions. Also the method has been described for one visible set of features selected on one side of the object, but it can be extended to all the sides that can appear in a working situation.
As will be apparent to those skilled in the art in the light of the foregoing disclosure, many alterations and modifications are possible in the practice of this invention without departing from the spirit or scope thereof.
Accordingly, the scope of the invention is to be construed in accordance with the substance defined by the following claims.
Claims
1. A method of three-dimensional handling of an object by a robot using a tool and one camera mounted on the robot, comprising:
i) calibrating the camera by fording a) the camera intrinsic parameter;
b) the position of the camera relative to the tool of the robot ( "hand - eye " calibration ) and c) the position of the camera in a space rigid to the place where the object will be trained;
ii) teaching the object features by a) putting the object in the "User Frame " space and capturing an image with the robot in the calibration position where the "Camera - > User Frame " transformation was calculated;
b) selecting at least one anchor feature from the image and at least other visible features;
c) calculating the 3D position of each feature in "User Frame"
using the height of the feature relative to the object base plane and the "Camera - > User Frame " transformation;
d) defining an "Object Space " aligned with the "User Frame "
but connected to the object and transposing the 3D coordinates of the features into that space;
e) computing the "Object Space -> Camera " transformation using the 3D position of the features inside this space and the position in the image;
f) defining an "Object Frame" inside "Object Space " to be used for teaching the handling path;
g) computing the Object Frame position and orientation in "Tool Frame" using the transformation from "Object Frame- > Cam-era " and "Camera -> Tool";
h) sending the "Object Frame " to the robot and training the handling path inside the "Object Frame";
iii) carrying out object finding and positioning by a) positioning the robot in a predefined position above the bin containing the target object; b) if no object is in the field of view, moving the robot until an anchor feature is found in the image; c) using the position and orientation of the anchor feature to compute the position of the remaining set of selected features; d) with the positions of features from the image and their correspondent position in "Object Space"as calculated in the training session, using the camera calibra-tion to compute the transformation between the "Object Space " and "Camera Space "; e) using the said transformation to calculate the movement of the robot to position the camera so that it "looks" orthogonal to the object; f) finding the "Object Space -> Camera Space" transformation in the same way as in step d); g) computing the object frame memorized at training using the found transformation and "Camera - > Tool " transformation; h) sending the commuted "Object Frame" to the robot; i) using the "Tool" position to define the frame in "Robot Space" and performing the trained handling on the object inside the "Robot Space".
i) calibrating the camera by fording a) the camera intrinsic parameter;
b) the position of the camera relative to the tool of the robot ( "hand - eye " calibration ) and c) the position of the camera in a space rigid to the place where the object will be trained;
ii) teaching the object features by a) putting the object in the "User Frame " space and capturing an image with the robot in the calibration position where the "Camera - > User Frame " transformation was calculated;
b) selecting at least one anchor feature from the image and at least other visible features;
c) calculating the 3D position of each feature in "User Frame"
using the height of the feature relative to the object base plane and the "Camera - > User Frame " transformation;
d) defining an "Object Space " aligned with the "User Frame "
but connected to the object and transposing the 3D coordinates of the features into that space;
e) computing the "Object Space -> Camera " transformation using the 3D position of the features inside this space and the position in the image;
f) defining an "Object Frame" inside "Object Space " to be used for teaching the handling path;
g) computing the Object Frame position and orientation in "Tool Frame" using the transformation from "Object Frame- > Cam-era " and "Camera -> Tool";
h) sending the "Object Frame " to the robot and training the handling path inside the "Object Frame";
iii) carrying out object finding and positioning by a) positioning the robot in a predefined position above the bin containing the target object; b) if no object is in the field of view, moving the robot until an anchor feature is found in the image; c) using the position and orientation of the anchor feature to compute the position of the remaining set of selected features; d) with the positions of features from the image and their correspondent position in "Object Space"as calculated in the training session, using the camera calibra-tion to compute the transformation between the "Object Space " and "Camera Space "; e) using the said transformation to calculate the movement of the robot to position the camera so that it "looks" orthogonal to the object; f) finding the "Object Space -> Camera Space" transformation in the same way as in step d); g) computing the object frame memorized at training using the found transformation and "Camera - > Tool " transformation; h) sending the commuted "Object Frame" to the robot; i) using the "Tool" position to define the frame in "Robot Space" and performing the trained handling on the object inside the "Robot Space".
Priority Applications (7)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CA002369845A CA2369845A1 (en) | 2002-01-31 | 2002-01-31 | Method and apparatus for single camera 3d vision guided robotics |
| US10/153,680 US6816755B2 (en) | 2002-01-31 | 2002-05-24 | Method and apparatus for single camera 3D vision guided robotics |
| PCT/CA2003/000139 WO2003064116A2 (en) | 2002-01-31 | 2003-01-31 | Method and apparatus for single camera 3d vision guided robotics |
| JP2003563782A JP2005515910A (en) | 2002-01-31 | 2003-01-31 | Method and apparatus for single camera 3D vision guide robotics |
| EP03734637A EP1472052A2 (en) | 2002-01-31 | 2003-01-31 | Method and apparatus for single camera 3d vision guided robotics |
| AU2003239171A AU2003239171A1 (en) | 2002-01-31 | 2003-01-31 | Method and apparatus for single camera 3d vision guided robotics |
| US10/634,874 US8095237B2 (en) | 2002-01-31 | 2003-08-06 | Method and apparatus for single image 3D vision guided robotics |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CA002369845A CA2369845A1 (en) | 2002-01-31 | 2002-01-31 | Method and apparatus for single camera 3d vision guided robotics |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CA2369845A1 true CA2369845A1 (en) | 2003-07-31 |
Family
ID=27587740
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA002369845A Abandoned CA2369845A1 (en) | 2002-01-31 | 2002-01-31 | Method and apparatus for single camera 3d vision guided robotics |
Country Status (2)
| Country | Link |
|---|---|
| US (1) | US6816755B2 (en) |
| CA (1) | CA2369845A1 (en) |
Families Citing this family (132)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2002172575A (en) * | 2000-12-07 | 2002-06-18 | Fanuc Ltd | Teaching device |
| US7065892B2 (en) * | 2001-03-19 | 2006-06-27 | Veeco Instruments Inc. | Method and apparatus for calibrating a vision system to a parts handling device |
| US20030048448A1 (en) * | 2001-03-19 | 2003-03-13 | Fleming Timothy J. | Automated apparatus for testing optical filters |
| US6983547B2 (en) * | 2001-03-19 | 2006-01-10 | Veeco Instruments Inc. | Goniometer |
| JP2003326486A (en) | 2001-06-20 | 2003-11-18 | Amada Co Ltd | Work positioning device |
| EP1472052A2 (en) * | 2002-01-31 | 2004-11-03 | Braintech Canada, Inc. | Method and apparatus for single camera 3d vision guided robotics |
| US7289230B2 (en) * | 2002-02-06 | 2007-10-30 | Cyberoptics Semiconductors, Inc. | Wireless substrate-like sensor |
| US6938454B2 (en) * | 2002-05-13 | 2005-09-06 | Trumpf Maschinen Austria Gmbh & Co. Kg. | Production device, especially a bending press, and method for operating said production device |
| AU2003257309A1 (en) | 2002-08-13 | 2004-02-25 | Microbotics Corporation | Microsurgical robot system |
| US7688381B2 (en) * | 2003-04-08 | 2010-03-30 | Vanbree Ken | System for accurately repositioning imaging devices |
| DE10345743A1 (en) * | 2003-10-01 | 2005-05-04 | Kuka Roboter Gmbh | Method and device for determining the position and orientation of an image receiving device |
| EP1524494A1 (en) * | 2003-10-17 | 2005-04-20 | inos Automationssoftware GmbH | Method for calibrating a camera-laser-unit in respect to a calibration-object |
| JP3905073B2 (en) * | 2003-10-31 | 2007-04-18 | ファナック株式会社 | Arc welding robot |
| JP2005138223A (en) * | 2003-11-06 | 2005-06-02 | Fanuc Ltd | Positional data correcting device for robot |
| JP3733364B2 (en) * | 2003-11-18 | 2006-01-11 | ファナック株式会社 | Teaching position correction method |
| DE102004005380A1 (en) | 2004-02-03 | 2005-09-01 | Isra Vision Systems Ag | Method for determining the position of an object in space |
| DE102004024378B4 (en) * | 2004-05-17 | 2009-05-20 | Kuka Roboter Gmbh | Method for robot-assisted measurement of objects |
| DE102004026185A1 (en) * | 2004-05-28 | 2005-12-22 | Kuka Roboter Gmbh | Method and apparatus for operating a machine, such as a multi-axis industrial robot |
| WO2006019970A2 (en) * | 2004-07-14 | 2006-02-23 | Braintech Canada, Inc. | Method and apparatus for machine-vision |
| US7640076B2 (en) * | 2004-10-29 | 2009-12-29 | Bradley Darrel Olson | Remote control rover configured for viewing and manipulating objects |
| DE102005007536A1 (en) * | 2005-02-17 | 2007-01-04 | Isra Vision Systems Ag | Method for calibrating a measuring system |
| US8108072B2 (en) * | 2007-09-30 | 2012-01-31 | Intuitive Surgical Operations, Inc. | Methods and systems for robotic instrument tool tracking with adaptive fusion of kinematics information and image information |
| US8147503B2 (en) * | 2007-09-30 | 2012-04-03 | Intuitive Surgical Operations Inc. | Methods of locating and tracking robotic instruments in robotic surgical systems |
| US8073528B2 (en) * | 2007-09-30 | 2011-12-06 | Intuitive Surgical Operations, Inc. | Tool tracking systems, methods and computer products for image guided surgery |
| US10555775B2 (en) | 2005-05-16 | 2020-02-11 | Intuitive Surgical Operations, Inc. | Methods and system for performing 3-D tool tracking by fusion of sensor and/or camera derived data during minimally invasive robotic surgery |
| US8111904B2 (en) * | 2005-10-07 | 2012-02-07 | Cognex Technology And Investment Corp. | Methods and apparatus for practical 3D vision system |
| US7804306B2 (en) | 2006-02-21 | 2010-09-28 | CyterOptics Semiconductor, Inc. | Capacitive distance sensing in semiconductor processing tools |
| US7893697B2 (en) | 2006-02-21 | 2011-02-22 | Cyberoptics Semiconductor, Inc. | Capacitive distance sensing in semiconductor processing tools |
| US20070276539A1 (en) * | 2006-05-25 | 2007-11-29 | Babak Habibi | System and method of robotically engaging an object |
| US8560047B2 (en) | 2006-06-16 | 2013-10-15 | Board Of Regents Of The University Of Nebraska | Method and apparatus for computer aided surgery |
| US8437535B2 (en) * | 2006-09-19 | 2013-05-07 | Roboticvisiontech Llc | System and method of determining object pose |
| GB2455006A (en) | 2006-09-29 | 2009-05-27 | Cyberoptics Semiconductor Inc | Substrate-like particle sensor |
| JP2008296330A (en) * | 2007-05-31 | 2008-12-11 | Fanuc Ltd | Robot simulation device |
| US7957583B2 (en) * | 2007-08-02 | 2011-06-07 | Roboticvisiontech Llc | System and method of three-dimensional pose estimation |
| US8082064B2 (en) | 2007-08-24 | 2011-12-20 | Elite Engineering Corporation | Robotic arm and control system |
| EP2048599B1 (en) * | 2007-10-11 | 2009-12-16 | MVTec Software GmbH | System and method for 3D object recognition |
| JP4347386B2 (en) * | 2008-01-23 | 2009-10-21 | ファナック株式会社 | Processing robot program creation device |
| US9333654B2 (en) * | 2008-03-31 | 2016-05-10 | Abb Research Ltd. | Robot parts assembly on a workpiece moving on an assembly line |
| US8238639B2 (en) | 2008-04-09 | 2012-08-07 | Cognex Corporation | Method and system for dynamic feature detection |
| JP2009269134A (en) * | 2008-05-08 | 2009-11-19 | Denso Wave Inc | Simulation device in visual inspection apparatus |
| US8180487B1 (en) | 2008-09-30 | 2012-05-15 | Western Digital Technologies, Inc. | Calibrated vision based robotic system |
| US8559699B2 (en) | 2008-10-10 | 2013-10-15 | Roboticvisiontech Llc | Methods and apparatus to facilitate operations in image based systems |
| US8135208B1 (en) | 2009-01-15 | 2012-03-13 | Western Digital Technologies, Inc. | Calibrated vision based robotic system utilizing upward and downward looking cameras |
| JP5981143B2 (en) | 2009-02-03 | 2016-08-31 | ファナック アメリカ コーポレイション | Robot tool control method |
| JP2011000703A (en) | 2009-05-19 | 2011-01-06 | Canon Inc | Manipulator with camera |
| US8219274B2 (en) * | 2009-07-02 | 2012-07-10 | Robert Bosch Gmbh | 3-dimensional perception system and method for mobile platform |
| JP4763074B2 (en) | 2009-08-03 | 2011-08-31 | ファナック株式会社 | Measuring device and measuring method of position of tool tip of robot |
| JP2011104906A (en) * | 2009-11-18 | 2011-06-02 | Mitsubishi Heavy Ind Ltd | Inspection method, method for producing composite material parts, inspection apparatus and apparatus for producing composite material parts |
| DE102010010718A1 (en) * | 2010-03-09 | 2011-09-15 | Kuka Laboratories Gmbh | Method for assembling components by means of an industrial robot |
| EP2385483B1 (en) | 2010-05-07 | 2012-11-21 | MVTec Software GmbH | Recognition and pose determination of 3D objects in 3D scenes using geometric point pair descriptors and the generalized Hough Transform |
| JP5682810B2 (en) * | 2010-09-15 | 2015-03-11 | セイコーエプソン株式会社 | robot |
| JP5895337B2 (en) | 2010-09-15 | 2016-03-30 | セイコーエプソン株式会社 | robot |
| US8781629B2 (en) | 2010-09-22 | 2014-07-15 | Toyota Motor Engineering & Manufacturing North America, Inc. | Human-robot interface apparatuses and methods of controlling robots |
| WO2012142587A1 (en) | 2011-04-15 | 2012-10-18 | Irobot Corporation | Method for path generation for an end effector of a robot |
| US11911117B2 (en) | 2011-06-27 | 2024-02-27 | Board Of Regents Of The University Of Nebraska | On-board tool tracking system and methods of computer assisted surgery |
| US9498231B2 (en) | 2011-06-27 | 2016-11-22 | Board Of Regents Of The University Of Nebraska | On-board tool tracking system and methods of computer assisted surgery |
| CN103764061B (en) | 2011-06-27 | 2017-03-08 | 内布拉斯加大学评议会 | On Tool Tracking System and Computer Assisted Surgery Method |
| US9279661B2 (en) * | 2011-07-08 | 2016-03-08 | Canon Kabushiki Kaisha | Information processing apparatus and information processing method |
| US8768071B2 (en) | 2011-08-02 | 2014-07-01 | Toyota Motor Engineering & Manufacturing North America, Inc. | Object category recognition methods and robots utilizing the same |
| US8755925B2 (en) | 2011-11-18 | 2014-06-17 | Nike, Inc. | Automated identification and assembly of shoe parts |
| US9451810B2 (en) | 2011-11-18 | 2016-09-27 | Nike, Inc. | Automated identification of shoe parts |
| US8849620B2 (en) | 2011-11-18 | 2014-09-30 | Nike, Inc. | Automated 3-D modeling of shoe parts |
| US8958901B2 (en) | 2011-11-18 | 2015-02-17 | Nike, Inc. | Automated manufacturing of shoe parts |
| US10552551B2 (en) | 2011-11-18 | 2020-02-04 | Nike, Inc. | Generation of tool paths for shore assembly |
| JP5910725B2 (en) * | 2012-02-27 | 2016-04-27 | 株式会社安川電機 | Robot system |
| JP2013184257A (en) | 2012-03-08 | 2013-09-19 | Sony Corp | Robot apparatus, method for controlling robot apparatus, and computer program |
| EP2685403B1 (en) | 2012-07-09 | 2025-04-23 | Deep Learning Robotics Ltd. | Natural machine interface system |
| US9221137B2 (en) * | 2012-10-05 | 2015-12-29 | Beckman Coulter, Inc. | System and method for laser-based auto-alignment |
| US20140100694A1 (en) * | 2012-10-05 | 2014-04-10 | Beckman Coulter, Inc. | System and method for camera-based auto-alignment |
| US10105149B2 (en) | 2013-03-15 | 2018-10-23 | Board Of Regents Of The University Of Nebraska | On-board tool tracking system and methods of computer assisted surgery |
| ES2522921B2 (en) | 2013-05-17 | 2015-07-30 | Loxin 2002, S.L. | Head and automatic machining procedure with vision |
| US9555549B2 (en) * | 2013-10-31 | 2017-01-31 | Seiko Epson Corporation | Control device, robot, robot system, and control method |
| JP6005299B2 (en) * | 2013-11-28 | 2016-10-12 | 三菱電機株式会社 | Robot system and control method of robot system |
| JP5850958B2 (en) * | 2014-01-24 | 2016-02-03 | ファナック株式会社 | Robot programming device for creating a robot program for imaging a workpiece |
| US10310054B2 (en) * | 2014-03-21 | 2019-06-04 | The Boeing Company | Relative object localization process for local positioning system |
| JP5778311B1 (en) * | 2014-05-08 | 2015-09-16 | 東芝機械株式会社 | Picking apparatus and picking method |
| US9734401B2 (en) | 2014-08-08 | 2017-08-15 | Roboticvisiontech, Inc. | Detection and tracking of item features |
| US9193073B1 (en) * | 2014-10-15 | 2015-11-24 | Quanta Storage Inc. | Robot calibration apparatus for calibrating a robot arm |
| TWI577493B (en) | 2014-12-26 | 2017-04-11 | 財團法人工業技術研究院 | Calibration method and automatic apparatus using the same |
| US9757859B1 (en) * | 2016-01-21 | 2017-09-12 | X Development Llc | Tooltip stabilization |
| US9744665B1 (en) | 2016-01-27 | 2017-08-29 | X Development Llc | Optimization of observer robot locations |
| US10059003B1 (en) | 2016-01-28 | 2018-08-28 | X Development Llc | Multi-resolution localization system |
| US9868212B1 (en) * | 2016-02-18 | 2018-01-16 | X Development Llc | Methods and apparatus for determining the pose of an object based on point cloud data |
| US10238571B2 (en) | 2016-06-22 | 2019-03-26 | Toyota Motor Engineering & Manufacturing North America, Inc. | Vision-assist devices and methods of calibrating image data of a vision-assist device |
| JP6500852B2 (en) * | 2016-07-11 | 2019-04-17 | 株式会社安川電機 | Robot system, robot control method, robot controller |
| US10078908B2 (en) | 2016-08-12 | 2018-09-18 | Elite Robotics | Determination of relative positions |
| JP6705738B2 (en) | 2016-12-05 | 2020-06-03 | 株式会社ソニー・インタラクティブエンタテインメント | Information processing apparatus, information processing method, and program |
| JP7051287B2 (en) * | 2016-12-05 | 2022-04-11 | 株式会社ソニー・インタラクティブエンタテインメント | Systems, jigs, and information processing methods |
| JP6922204B2 (en) * | 2016-12-09 | 2021-08-18 | セイコーエプソン株式会社 | Controls, robots and robot systems |
| KR102576842B1 (en) * | 2017-01-04 | 2023-09-12 | 삼성전자주식회사 | Robot and electronic device for performing hand-eye calibration |
| US10882185B2 (en) | 2017-02-07 | 2021-01-05 | Veo Robotics, Inc. | Dynamically determining workspace safe zones with speed and separation monitoring |
| US12103170B2 (en) | 2017-01-13 | 2024-10-01 | Clara Vu | Dynamic, interactive signaling of safety-related conditions in a monitored environment |
| US11518051B2 (en) | 2017-02-07 | 2022-12-06 | Veo Robotics, Inc. | Dynamic, interactive signaling of safety-related conditions in a monitored environment |
| US11820025B2 (en) | 2017-02-07 | 2023-11-21 | Veo Robotics, Inc. | Safe motion planning for machinery operation |
| US11541543B2 (en) | 2017-02-07 | 2023-01-03 | Veo Robotics, Inc. | Dynamic, interactive signaling of safety-related conditions in a monitored environment |
| US10579890B2 (en) * | 2017-02-28 | 2020-03-03 | Quality Vision International Inc. | Automatic alignment of a 3D model to a test object |
| JP2018167334A (en) * | 2017-03-29 | 2018-11-01 | セイコーエプソン株式会社 | Teaching device and teaching method |
| DE102017005194C5 (en) | 2017-05-31 | 2022-05-19 | Kuka Deutschland Gmbh | Controlling a robot assembly |
| CN107590835B (en) * | 2017-08-24 | 2020-12-25 | 中国东方电气集团有限公司 | Mechanical arm tool quick-change visual positioning system and positioning method in nuclear environment |
| US11072074B2 (en) * | 2017-12-13 | 2021-07-27 | Cognex Corporation | Calibration and operation of vision-based manipulation systems |
| US12049014B2 (en) | 2018-02-06 | 2024-07-30 | Veo Robotics, Inc. | Workplace monitoring and semantic entity identification for safe machine operation |
| US12097625B2 (en) | 2018-02-06 | 2024-09-24 | Veo Robotics, Inc. | Robot end-effector sensing and identification |
| US11745354B2 (en) | 2018-08-16 | 2023-09-05 | Mitutoyo Corporation | Supplementary metrology position coordinates determination system including an alignment sensor for use with a robot |
| US11002529B2 (en) | 2018-08-16 | 2021-05-11 | Mitutoyo Corporation | Robot system with supplementary metrology position determination system |
| US10871366B2 (en) | 2018-08-16 | 2020-12-22 | Mitutoyo Corporation | Supplementary metrology position coordinates determination system for use with a robot |
| US10751883B2 (en) * | 2018-08-16 | 2020-08-25 | Mitutoyo Corporation | Robot system with supplementary metrology position coordinates determination system |
| US10913156B2 (en) | 2018-09-24 | 2021-02-09 | Mitutoyo Corporation | Robot system with end tool metrology position coordinates determination system |
| US10926416B2 (en) * | 2018-11-21 | 2021-02-23 | Ford Global Technologies, Llc | Robotic manipulation using an independently actuated vision system, an adversarial control scheme, and a multi-tasking deep learning architecture |
| US11911914B2 (en) | 2019-01-28 | 2024-02-27 | Cognex Corporation | System and method for automatic hand-eye calibration of vision system for robot motion |
| US12036663B2 (en) * | 2019-03-25 | 2024-07-16 | Abb Schweiz Ag | Method and control arrangement for determining a relation between a robot coordinate system and a movable apparatus coordinate system |
| US10906184B2 (en) | 2019-03-29 | 2021-02-02 | Mujin, Inc. | Method and control system for verifying and updating camera calibration for robot control |
| CN110407010B (en) * | 2019-06-25 | 2021-03-19 | 安吉八塔机器人有限公司 | A handling and rectifying system suitable for elastic materials and method thereof |
| EP3988254A4 (en) * | 2019-07-19 | 2023-02-01 | Siemens Ltd., China | METHOD AND DEVICE FOR ROBOT HAND-EYE CALIBRATION, COMPUTING DEVICE, MEDIA AND PRODUCT |
| CN110763152B (en) * | 2019-10-09 | 2021-08-20 | 哈尔滨工程大学 | An underwater active rotating structured light three-dimensional vision measurement device and measurement method |
| US11173610B2 (en) * | 2019-11-13 | 2021-11-16 | Vicarious Fpc, Inc. | Method and system for robot control using visual feedback |
| CN112123329B (en) * | 2020-02-21 | 2024-09-13 | 深圳市三宝创新智能有限公司 | A robot 3D vision hand-eye calibration method |
| US11676845B2 (en) | 2020-06-30 | 2023-06-13 | Brooks Automation Us, Llc | Automated teach apparatus for robotic systems and method therefor |
| US11969218B2 (en) * | 2020-07-05 | 2024-04-30 | Asensus Surgical Us, Inc. | Augmented reality surgery set-up for robotic surgical procedures |
| CN112165562B (en) * | 2020-08-27 | 2023-04-14 | 深圳玖加科技有限公司 | Object photographing method and device in carrying process |
| CN112140045A (en) * | 2020-08-28 | 2020-12-29 | 国网安徽省电力有限公司淮南供电公司 | Electric wrench of working robot based on visual guidance and control method thereof |
| CN112525074B (en) * | 2020-11-24 | 2022-04-12 | 杭州素问九州医疗科技有限公司 | Calibration method, calibration system, robot, computer device and navigation system |
| CN113070876A (en) * | 2021-03-19 | 2021-07-06 | 深圳群宾精密工业有限公司 | Manipulator dispensing path guiding and deviation rectifying method based on 3D vision |
| DE102021203779B4 (en) * | 2021-04-16 | 2023-12-14 | Volkswagen Aktiengesellschaft | Method and device for annotating images of an object recorded with the aid of a camera |
| TWI782709B (en) * | 2021-09-16 | 2022-11-01 | 財團法人金屬工業研究發展中心 | Surgical robotic arm control system and surgical robotic arm control method |
| KR102651649B1 (en) * | 2021-11-23 | 2024-03-26 | 세메스 주식회사 | Substrate Treating Apparatus and Substrate Treating Method Using The Same |
| US12174005B2 (en) | 2021-12-27 | 2024-12-24 | Mitutoyo Corporation | Metrology system with position and orientation tracking utilizing light beams |
| CN114536340A (en) * | 2022-03-11 | 2022-05-27 | 南通西塔自动化科技有限公司 | Automatic grabbing method and system for iron roughneck based on machine vision assistance |
| CN114434059B (en) * | 2022-04-08 | 2022-07-01 | 西安知象光电科技有限公司 | Automatic welding system and method for large structural part with combined robot and three-dimensional vision |
| CN114833825B (en) * | 2022-04-19 | 2024-06-14 | 深圳市大族机器人有限公司 | Collaborative robot control method, device, computer equipment and storage medium |
| TWI799310B (en) * | 2022-06-30 | 2023-04-11 | 國立清華大學 | Robot and robot hand-eye callibrating method |
| US20240265565A1 (en) * | 2023-02-06 | 2024-08-08 | Walmart Apollo, Llc | Systems and methods for analyzing depth in images obtained in product storage facilities to detect outlier items |
| CN118952289B (en) * | 2024-10-11 | 2025-01-07 | 北京小米机器人技术有限公司 | Method for acquiring tail end operation data of mechanical arm through data acquisition equipment |
Family Cites Families (25)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US3986007A (en) | 1975-08-20 | 1976-10-12 | The Bendix Corporation | Method and apparatus for calibrating mechanical-visual part manipulating system |
| US4146924A (en) | 1975-09-22 | 1979-03-27 | Board Of Regents For Education Of The State Of Rhode Island | System for visually determining position in space and/or orientation in space and apparatus employing same |
| CA1103803A (en) | 1978-03-01 | 1981-06-23 | National Research Council Of Canada | Method and apparatus of determining the center of area or centroid of a geometrical area of unspecified shape lying in a larger x-y scan field |
| JPS5923467B2 (en) | 1979-04-16 | 1984-06-02 | 株式会社日立製作所 | Position detection method |
| US4373804A (en) | 1979-04-30 | 1983-02-15 | Diffracto Ltd. | Method and apparatus for electro-optically determining the dimension, location and attitude of objects |
| US4305130A (en) | 1979-05-29 | 1981-12-08 | University Of Rhode Island | Apparatus and method to enable a robot with vision to acquire, orient and transport workpieces |
| US6317953B1 (en) | 1981-05-11 | 2001-11-20 | Lmi-Diffracto | Vision target based assembly |
| US5506682A (en) | 1982-02-16 | 1996-04-09 | Sensor Adaptive Machines Inc. | Robot vision using targets |
| US4654949A (en) | 1982-02-16 | 1987-04-07 | Diffracto Ltd. | Method for automatically handling, assembling and working on objects |
| US4613942A (en) | 1982-02-19 | 1986-09-23 | Chen Richard M | Orientation and control system for robots |
| US4437114A (en) | 1982-06-07 | 1984-03-13 | Farrand Optical Co., Inc. | Robotic vision system |
| EP0151417A1 (en) | 1984-01-19 | 1985-08-14 | Hitachi, Ltd. | Method for correcting systems of coordinates in a robot having visual sensor device and apparatus therefor |
| US4578561A (en) * | 1984-08-16 | 1986-03-25 | General Electric Company | Method of enhancing weld pool boundary definition |
| US5267143A (en) | 1984-10-12 | 1993-11-30 | Sensor Adaptive Machines, Incorporated | Vision assisted fixture construction |
| JPS63288683A (en) * | 1987-05-21 | 1988-11-25 | 株式会社東芝 | Assembling robot |
| JPH01124072A (en) * | 1987-11-09 | 1989-05-16 | Seiko Instr & Electron Ltd | Pick and place |
| JP2710850B2 (en) | 1989-03-27 | 1998-02-10 | キヤノン株式会社 | Work holding device, work and its storage case |
| JP2686351B2 (en) | 1990-07-19 | 1997-12-08 | ファナック株式会社 | Vision sensor calibration method |
| US5083073A (en) * | 1990-09-20 | 1992-01-21 | Mazada Motor Manufacturing U.S.A. Corp. | Method and apparatus for calibrating a vision guided robot |
| US5454775A (en) * | 1994-09-13 | 1995-10-03 | Applied Robotics, Inc. | Automated exchangeable parts feeding system |
| EP0884141B1 (en) | 1996-10-24 | 2006-07-19 | Fanuc Ltd | Force control robot system with visual sensor for inserting work |
| JPH11300670A (en) | 1998-04-21 | 1999-11-02 | Fanuc Ltd | Article picking-up device |
| US5959425A (en) | 1998-10-15 | 1999-09-28 | Fanuc Robotics North America, Inc. | Vision guided automatic robotic path teaching method |
| US6341246B1 (en) * | 1999-03-26 | 2002-01-22 | Kuka Development Laboratories, Inc. | Object oriented motion system |
| US6466843B1 (en) * | 2001-10-16 | 2002-10-15 | General Electric Company | Method and apparatus for lifting objects |
-
2002
- 2002-01-31 CA CA002369845A patent/CA2369845A1/en not_active Abandoned
- 2002-05-24 US US10/153,680 patent/US6816755B2/en not_active Expired - Lifetime
Also Published As
| Publication number | Publication date |
|---|---|
| US20030144765A1 (en) | 2003-07-31 |
| US6816755B2 (en) | 2004-11-09 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA2369845A1 (en) | Method and apparatus for single camera 3d vision guided robotics | |
| US8095237B2 (en) | Method and apparatus for single image 3D vision guided robotics | |
| CN113524194B (en) | Target grabbing method of robot vision grabbing system based on multi-mode feature deep learning | |
| Chang | Robotic assembly of smartphone back shells with eye-in-hand visual servoing | |
| CN110136208B (en) | Joint automatic calibration method and device for robot vision servo system | |
| US20200061837A1 (en) | Method for industrial robot commissioning, industrial robot system and control system using the same | |
| JP2023052266A (en) | Systems and methods for combining machine vision coordinate spaces in a guided assembly environment | |
| US20200298411A1 (en) | Method for the orientation of an industrial robot, and industrial robot | |
| US20140118500A1 (en) | System and method for finding correspondence between cameras in a three-dimensional vision system | |
| CN113379849A (en) | Robot autonomous recognition intelligent grabbing method and system based on depth camera | |
| US11488322B2 (en) | System and method for training a model in a plurality of non-perspective cameras and determining 3D pose of an object at runtime with the same | |
| CN111612794A (en) | High-precision 3D pose estimation method and system for parts based on multi-2D vision | |
| Mittrapiyanumic et al. | Calculating the 3d-pose of rigid-objects using active appearance models | |
| Jia et al. | Real-time color-based sorting robotic arm system | |
| US20230150142A1 (en) | Device and method for training a machine learning model for generating descriptor images for images of objects | |
| Oh et al. | Development of structured light based bin picking system using primitive models | |
| Erceg et al. | Stereo vision based robot welding | |
| CN114187312A (en) | Target object grabbing method, device, system, storage medium and equipment | |
| CN117656066A (en) | Calibration and material taking method and calibration system based on manipulator | |
| Motai et al. | SmartView: hand-eye robotic calibration for active viewpoint generation and object grasping | |
| JP7733108B2 (en) | Image capturing device for calculating three-dimensional position based on an image captured by a visual sensor | |
| Kao et al. | Object pose estimation and feature extraction based on PVNet | |
| WO2012076979A1 (en) | Model-based pose estimation using a non-perspective camera | |
| Zhao et al. | Using 3D matching for picking and placing on UR robot | |
| CN119672108B (en) | Coordinate reconstruction method, device and storage medium for coupling optical fiber panel and sensor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| EEER | Examination request | ||
| FZDE | Discontinued |