JP7576620B2 - How to measure camera position deviation - Google Patents

Description

This specification discloses a method for measuring camera position deviation.

Conventionally, a robot equipped with an arm has been proposed that calibrates a camera installed independently of the arm (see, for example, Patent Document 1). The procedure for calibrating the camera involves first setting three rotation axes X, Y, and Z of the hand coordinate system, and operating the arm so as to rotate a calibration pattern held at the hand around each rotation axis. Next, the camera captures pattern images of the calibration pattern at multiple rotation positions around each rotation axis. These pattern images are then used to estimate a coordinate transformation matrix between the hand coordinate system and the camera coordinate system. According to this procedure, external parameters that can be used to calculate the coordinate transformation between the hand coordinate system and the camera coordinate system are obtained, making it possible to detect the position of an object using a camera.

JP 2019-14031 A

Although the above-mentioned Patent Document 1 describes the calibration of a camera installed independently of an arm, it does not make any mention of calibrating a camera attached to an arm (measuring positional deviation).

The primary objective of this disclosure is to measure the positional deviation of a camera attached to an arm using a simple method.

The camera position shift measurement method disclosed herein employs the following measures to achieve the above-mentioned primary objective.

The camera position deviation amount measuring method disclosed herein includes:
1. A method for measuring a positional deviation of a camera in a robot having a robot arm, a hand attached to a tip of the robot arm via a rotation shaft, and a camera attached to the robot arm such that an optical axis of the camera is parallel to the rotation shaft, comprising:
(a) attaching a jig having a mark to the hand;
(b) capturing images of the mark with the camera at a plurality of rotation positions while rotating the hand around the rotation axis;
a step (c) of calculating a positional deviation of the camera based on a difference between a position of a mark shown in each captured image taken at each of the plurality of rotational positions and an ideal position of the mark;
The gist of the invention is to provide the following:

The camera misalignment measurement method disclosed herein measures the misalignment of a camera in a robot having a robot arm, a hand attached to the tip of the robot arm via a rotation axis, and a camera attached to the robot arm so that its optical axis is parallel to the rotation axis. In this method, a jig with a mark is attached to the hand, and the mark is captured by the camera at multiple rotation positions while the hand is rotated around the rotation axis. The camera misalignment is then calculated based on the difference between the position of the mark in each captured image taken at each of the multiple rotation positions and the ideal position of the mark. This allows the measurement of the misalignment of the camera attached to the arm to be performed using a simple procedure.

FIG. 2 is an external perspective view of the working robot. FIG. FIG. 2 is a block diagram showing an electrical connection relationship between a robot body and a control device. FIG. 11 is an explanatory diagram showing an example of a camera position deviation measurement procedure. FIG. 13 is an explanatory diagram showing how a jig with a mark is attached. 1 is an explanatory diagram showing the designed camera field of view and the actual camera field of view when a positional deviation occurs in the camera; FIG. 13 is an explanatory diagram for explaining a positional deviation (x _c , y _c ) and a rotational deviation θ _c in an _{Xvr Yvr} coordinate system based on an actual camera field of view. FIG. 13 is an explanatory diagram for explaining a positional deviation (x _ci , y _ci ) in an X _vi Y _vi coordinate system based on a designed camera field of view.

Next, the form for implementing the present disclosure will be described with reference to the drawings.

Figure 1 is an external perspective view of the work robot. Figure 2 is a side view of the robot body. Figure 3 is a block diagram showing the electrical connection relationship between the robot body and a control device.

The working robot 1 is configured as a horizontal articulated robot that performs a predetermined task on a workpiece W (for example, a transport task of picking up and transporting the workpiece W, or an assembly task of picking up the workpiece W and assembling it to an object). The working robot 1 comprises a robot body 10 (see Figures 1 to 3) and a control device 70 (see Figure 3) that controls the robot body 10. As shown in Figures 1 and 2, the robot body 10 comprises a base 11 and an articulated arm 20.

The base 11 is fixed to the workbench 2 and supports the base end side of the multi-joint arm 20. The multi-joint arm 20 includes a first arm 21, a first arm drive unit 30, a second arm 22, a second arm drive unit 40, a shaft 23, a shaft drive unit 50, and a camera 60. The base end of the first arm 21 is connected to the base 11 via a first joint axis J1, and is configured to be rotatable (horizontally rotated) in a horizontal plane relative to the base 11 by the rotation of the first joint axis J1. The base end of the second arm 22 is connected to the tip of the first arm 21 via a second joint axis J2, and is configured to be rotatable (horizontally rotated) in a horizontal plane relative to the first arm 21 by the rotation of the second joint axis J2. The shaft 23 is connected to the tip of the second arm 22 via a third joint axis J3, and is configured to be rotatable around the axis of the third joint axis J3 relative to the second arm 22 and to be raised and lowered along the axial direction of the third joint axis J3. At the tip of the shaft 23, a tool holding portion 24 for holding various tools for performing operations on the workpiece W is provided.

3, the first arm driving unit 30 includes a motor 32 and an encoder 34. The rotating shaft of the motor 32 is connected to the first joint shaft J1 via a reducer (not shown). The first arm driving unit 30 drives the motor 32 to transmit torque to the first joint shaft J1 via the reducer, causing the first arm 21 to rotate horizontally around the first joint shaft J1 as a fulcrum. The encoder 34 is attached to the rotating shaft of the motor 32 and is configured as a rotary encoder that detects the amount of rotational displacement of the motor 32.

The second arm driving unit 40, like the first arm driving unit 30, includes a motor 42 and an encoder 44. The rotating shaft of the motor 42 is connected to the second joint shaft J2 via a reduction gear (not shown). The second arm driving unit 40 drives the motor 42 to transmit torque to the second joint shaft J2 via the reduction gear, causing the second arm 22 to rotate horizontally around the second joint shaft J2 as a fulcrum. The encoder 44 is attached to the rotating shaft of the motor 42 and is configured as a rotary encoder that detects the amount of rotational displacement of the motor 42.

As shown in FIG. 3, the shaft drive unit 50 includes motors 52a and 52b and encoders 54a and 54b. The rotating shaft of the motor 52a is connected to the shaft 23 via a belt (not shown) and rotates the shaft 23 around its axis. The rotating shaft of the motor 52b is connected to a ball screw nut (not shown) that passes through the shaft 23 via a belt and raises and lowers the shaft 23 by rotating the ball screw nut. The encoder 54a is configured as a rotary encoder that detects the amount of rotational displacement of the shaft 23. The encoder 54b is configured as a linear encoder that detects the raised and lowered position of the shaft 23.

The camera 60 is attached to the side of the tip of the second arm 22 so that its optical axis is parallel to the axis of the shaft 23. The camera 60 captures an image of the workpiece W to be worked on and outputs the captured image to the control device 70. The control device 70 recognizes the position of the workpiece W by processing the captured image.

3, the control device 70 includes a CPU 71, a ROM 72, a RAM 73, a non-volatile memory (storage device 74), and an input/output interface (not shown). Position signals from the encoders 34, 44, 54a, and 54b and image signals from the camera 60 are input to the control device 70 via the input/output interface. Drive signals for the motors 32, 42, 52a, and 52b are output from the control device 70 via the input/output interface.

Next, the operation of the working robot 1 thus configured will be described. The CPU 71 of the control device 70 first controls the first arm drive unit 30, the second arm drive unit 40, and the shaft drive unit 50 so that the hand moves to a predetermined target position in order to move the camera 60 above the workpiece W, and captures the workpiece W with the camera 60. Next, the CPU 71 performs image processing to process the captured image and measures the position of the workpiece W in the camera coordinate system. The image processing is performed by measuring the coordinate values of the workpiece W shown in the captured image, and performing position offset correction and rotation offset correction on the measured coordinate values using the correction values. Here, the correction values are used to correct the deviation in the positional relationship between the camera 60 and the hand caused by processing errors, assembly errors, etc. of the camera 60. The correction values are measured in advance by a camera position deviation measurement procedure described later, and stored in the storage device 74. Next, the CPU 71 converts the position of the workpiece W into the robot coordinate system, and sets the target position of the hand for picking up the workpiece W based on the converted position of the workpiece W. Then, the CPU 71 controls the first arm driving unit 30, the second arm driving unit 40, and the shaft driving unit 50 so that the end effector moves to the set target position. The workpiece W is picked up by the end effector (workpiece holding unit 24) as the end effector moves to the target position.

Next, the camera position deviation measurement procedure will be described. FIG. 4 is an explanatory diagram showing an example of the camera position deviation measurement procedure. The camera position deviation measurement procedure is executed by steps S100 to S150.

In step S100, as shown in FIG. 5, the marked jig 100 is attached to the tool holding unit 24 (hand tip) provided at the tip of the shaft 23. The marked jig 100 has a mark M at a position spaced apart from the attachment position on the tool holding unit 24 in a direction perpendicular to the shaft 23. In step S110, the shaft driving unit 50 is controlled so that the shaft 23 rotates at a predetermined angle (for example, 10 degrees), and the camera 60 captures the mark M every time the shaft 23 rotates at the predetermined angle. As a result, captured images of the mark M are obtained at a plurality of rotation positions n (n=1, 2, 3 , ... , k ). In step S120, the mark position ( _xvn , _yvn ) (n=1, 2, 3, ..., k) for each rotation position n is measured in a coordinate system ( _XvrYvr coordinate _system ) based on the actual camera field of view including the positional deviation of the camera 60 from the captured image . The mark position (x _vn , y _vn ) is measured by processing the captured image and determining the central coordinates of the mark M shown in the image.

Step S130 derives the positional deviation ( _xc , _yc ) and rotational deviation θc of the camera 60 based on the difference between the mark position ( _xvn , _yvn ) (n=1, 2, 3 , ..., k ) measured in _{the XvrYvr} coordinate system and the ideal mark position ( _xin , _yin ) (n=1, 2, 3 , ... , k ) stored in advance in the ROM 72 in the XvrYvr coordinate system. The positional deviation ( _xc , _yc ) indicates the _amount of positional deviation in the _Xvr axis direction and the amount of positional deviation in the _Yvr axis direction _{, respectively. Furthermore, the rotational deviation θc indicates} the amount of deviation in the rotational _direction of the actual camera field of view including the positional deviation of the camera 60 with respect to the designed camera field of view not including the positional deviation of the camera 60. In this embodiment, step S130 is performed by deriving a combination of positional deviation (xc, yc) and rotational deviation _θc that minimizes the sum of squares _{F of the difference fxn in} the _Xvr axis direction and the difference fyn in the _Yvr axis direction defined by the following expressions ( ₁₎ to ( ₃ ), using a well-known minimization algorithm such as a genetic algorithm or a Newton-Raphson method. Step S140 converts the positional deviation ( _xc , _yc ) from a coordinate system based on the actual camera view ( _XvrYvr coordinate system ₎ to a coordinate system based on the designed camera view ( _XviYvi coordinate system) using the rotational deviation θc according to the following expressions ( _{4 ) and (5). As a result, the positional deviation (xci , yci )} of camera 60 in the _{XviYvi axis} direction is derived in the _XviYvi coordinate system. In step S150, a position offset value that is the same amount as the positional deviation (x _ci , y _ci ) but in the opposite direction, and a rotation offset value that is the same amount as the rotational deviation θ _c but in the opposite direction are registered in the storage device 74 as correction values. As a result, when picking up the workpiece W, the position of the workpiece W measured by capturing an image of the workpiece W with the camera 60 is offset corrected (position offset and rotation offset), so that the position of the workpiece W can be correctly recognized regardless of the positional deviation of the camera 60. As a result, the accuracy of the operation (pickup) can be further improved.

Here, in step S130, the following formulas (6) and (7) theoretically hold between the measured mark position ( _xvn , _yvn ) and the ideal mark position ( _xin , _yin ). For this reason, it is possible to derive the positional shift ( _xc , _yc ) and rotational shift _θc of the camera 60 using formulas (6) and (7). However, in reality, the relationship between the two contains various errors, so the correction accuracy can be further improved by finding a combination of the positional shift ( _xc , _yc ) and rotational shift _θc that minimizes the sum of squares F with the differences _fxn , _fyn .

Here, we will explain the correspondence between the main elements of the embodiment and the main elements of the present disclosure described in the claims. That is, in this embodiment, the articulated arm 20 corresponds to a robot arm, the tool holding unit 24 corresponds to a hand, and the camera 60 corresponds to a camera.

It goes without saying that the present disclosure is in no way limited to the above-described embodiments, and may be implemented in various forms as long as they fall within the technical scope of the present disclosure.

For example, in the above-described embodiment, the camera position deviation measuring method of the present disclosure has been described as being applied to a horizontal articulated robot. However, the present disclosure is not limited to this, and the present invention can be applied to any robot configuration, such as a vertical articulated robot, as long as the robot has a robot arm, a hand attached to the tip of the robot arm via a rotation axis, and a camera attached to the robot arm so that the optical axis is parallel to the rotation axis.

As described above, the camera positional deviation measurement method disclosed herein is a camera positional deviation measurement method for measuring the positional deviation of a robot having a robot arm, a hand attached to the tip of the robot arm via a rotation axis, and a camera attached to the robot arm so that its optical axis is parallel to the rotation axis, and comprises the steps of: (a) attaching a jig having a mark to the hand; (b) capturing images of the mark with the camera at a plurality of rotational positions while rotating the hand around the rotation axis; and (c) determining the positional deviation of the camera based on the difference between the position of the mark appearing in each of the captured images taken at the plurality of rotational positions and an ideal mark position.

The camera misalignment measurement method disclosed herein measures the misalignment of a camera in a robot having a robot arm, a hand attached to the tip of the robot arm via a rotation axis, and a camera attached to the robot arm so that its optical axis is parallel to the rotation axis. In this method, a jig with a mark is attached to the hand, and the mark is captured by the camera at multiple rotation positions while the hand is rotated around the rotation axis. The camera misalignment is then calculated based on the difference between the position of the mark in each captured image taken at each of the multiple rotation positions and the ideal position of the mark. This allows the measurement of the misalignment of the camera attached to the arm to be performed using a simple procedure.

In the camera positional deviation measuring method disclosed herein, in _{an XvrYvr} coordinate system based on an actual camera field of view, when the position of the mark imaged and measured at each rotational position n (n=1, 2, . . .) in step (b) is ( _xvn , _yvn ), the ideal position of the mark at each rotational position n (n=1, 2, . . . ) is (xin, _yin ), the positional deviations of the camera in the _Xvr axis direction and the _Yvr axis direction are ( _xc , _yc ), and a rotational deviation in the rotation direction of the actual camera field of view with respect to the designed camera field of view is _θc , the positional deviation ( _xc , _{yc )} and the rotational deviation _θc are calculated using a relational expression that represents the relationship between the measured mark position ( _xvn , _yvn ) and the ideal mark position ( _xin , _{yin), and the coordinate system of the positional deviation (xc, yc ) is} calculated based on the rotational deviation _{θc .} The positional deviation of the camera may be calculated by converting _{the vr} coordinate system into an _{XviYvi coordinate} system based on the designed camera field of view. In this way, the positional deviation of the camera can be calculated by simple processing from the positions of the marks captured and measured by the cameras at each rotational position n.

In this case, the step (c) may involve using a predetermined minimization algorithm to find a combination of the positional shift ( _xc , _yc ) and the rotational shift _θc that minimizes the sum of squares F defined by equations (1) to (3), and converting the coordinate system of the positional shift ( _xc , _yc ) from the _XvrYvr coordinate system to the _{XviYvi coordinate} system by the following equations (4) and (5) to find the positional shift ( _xci , _yci ) in _{the XviYvi} coordinate system. In _this way, the measurement accuracy of the positional shift and rotational shift of the camera can be further improved.

In addition, in the camera position shift measuring method disclosed herein, the offset value corresponding to the position shift and rotation shift of the camera obtained in step (c) may be registered as a correction value for correcting the position of the object captured and measured by the camera. In this way, it is possible to accurately recognize the position of the object using the camera.

This disclosure can be used in the robot manufacturing industry, etc.

1 Working robot, 2 Work table, 10 Robot body, 11 Base, 20 Articulated arm, 21 First arm, 22 Second arm, 23 Shaft, 30 First arm drive unit, 32 Motor, 34 Encoder, 40 Second arm drive unit, 42 Motor, 44 Encoder, 50 Shaft drive unit, 52a, 52b Motor, 54a, 54b Encoder, 60 Camera, 70 Control device, 71 CPU, 72 ROM, 73 RAM, 74 Storage device, J1 First joint axis, J2 Second joint axis, J3 Third joint axis.

Claims (2)

1. A method for measuring a positional deviation of a camera in a robot having a robot arm, a hand attached to a tip of the robot arm via a rotation shaft, and a camera attached to the robot arm such that an optical axis of the camera is parallel to the rotation shaft, comprising:
(a) attaching a jig having a mark to the hand;
(b) capturing images of the mark on the jig with the camera at a plurality of rotation positions while rotating the hand around the rotation axis;
a step (c) of calculating a positional deviation of the camera based on a difference between a position of the mark shown in each captured image taken at each of the plurality of rotational positions and an ideal position of the mark at each of the plurality of rotational positions;
Equipped with
In the step (c), in an _{XvrYvr coordinate} system based on an actual camera field of view , when the position of the mark imaged and measured at each rotation position n (n=1, 2, ..., k) in the step (b) is (xvn _, yvn ₎ , the ideal position of the mark at each rotation position n (n=1, 2, ..., k) is (xin, yin), the positional deviations of the camera in the Xvr _axis direction _and the Yvr axis direction are (xc _, yc _{) ,} and _a rotational deviation in the rotation direction of the actual camera field of view with respect to the designed camera field of view is _θc , _the positional deviation ( _xc , _yc ) and the rotational deviation θc _are calculated using a relational expression that represents the relationship between the measured mark position (xvn, yvn) and _{the ideal} mark position (xin , _yin ) , and the coordinate system of the positional deviation (xc, yc) is calculated based _on the rotational _{deviation θc .} A positional deviation of the camera is calculated by converting _{the vr} coordinate system into an XviYvi _{coordinate system} based on the designed camera field of view ,
A combination of the positional deviation (x _c , y _c ) and the rotational deviation θ _c that minimizes the sum of squares F defined by the following equations (1) to (3) is obtained using a predetermined minimization algorithm, and the coordinate system of the positional deviation (x _c , y _c ) is transformed from _{the XvrYvr} coordinate system to the XviYvi _coordinate system by the following equations ( ₄ ) and (5) to obtain the positional deviation (x _ci , y ci ₎ in _{the XviYvi} coordinate system ;
How to measure camera position deviation.

2. The camera position deviation measuring method according to claim 1 ,
a step (d) of registering an offset value corresponding to the positional and rotational deviations of the camera obtained in the step (c) as a correction value for correcting a position of an object imaged and measured by the camera.

Images