WO2018173243A1

WO2018173243A1 - Stereo camera device

Info

Publication number: WO2018173243A1
Application number: PCT/JP2017/011909
Authority: WO
Inventors: 媛李; 三好　雅則; 秀行粂
Original assignee: 株式会社日立製作所
Priority date: 2017-03-24
Filing date: 2017-03-24
Publication date: 2018-09-27

Abstract

The purpose of the present invention is to provide a highly reliable stereo camera device with consideration given to a blind spot region. To solve the above problem, this stereo camera device comprises: a plurality of real video input units 10 for photographing an object or a region; a camera control unit 12 for changing a set value at the time of photographing by the real video input unit 10; an image acquisition unit 11 for converting an electronic signal inputted from the real video input unit 10 into an image; a monitoring unit 13 for detecting abnormality and danger on the basis of image information acquired by the image acquisition unit 11; a stereo control unit 14 for calculating a control value 20 at the time of conversion to stereo of the real video input unit 10 on the basis of information detected by the monitoring unit 13; and a distance calculation unit 15 for calculating the distance from the real video input unit 10 to the object or the region on the basis of the information acquired by the image acquisition unit 11, and is characterized in that the camera control unit 12 changes the set value at the time of photographing by the real video input unit 10 on the basis of the control value 20 calculated by the stereo control unit 14.

Description

Stereo camera device

The present invention relates to a monocular camera or a stereo camera device having a plurality of cameras.

Conventionally, a stereo measurement apparatus that measures an object or an environment using a plurality of cameras has been put into practical use. For example, for the purpose of intruder detection and danger prevention, control is performed by various measurements using an in-vehicle stereo camera of an automobile. In addition, there is an example in which the vehicle is controlled by measuring the front-rear direction from a camera even in automatic driving of a car. In recent years, it is also an example of the application of stereo cameras in the surveillance field and marketing. As stereo three-dimensional measurement, two cameras photograph the same object at the same time, perform distortion correction and parallelization on the obtained image, and calculate a difference and parallax on the image. And the shape of a three-dimensional object etc. can be measured using the calculated parallax. Here, it is a condition that the measurement areas of the two cameras are substantially the same. In that case, the measurement range is fixed depending on specifications such as camera parameters, field angle, and parallax depth. For example, in the case of an in-vehicle stereo camera, the measurement range is defined from 2 m to 40 m in front of the camera. That is, there is a problem that it becomes impossible to measure 2 m or less and 40 m or more from the front of the camera.

Here, there is one described in Patent Document 1 as a means having a high ability to detect the surrounding situation. In Japanese Patent Laid-Open No. 2004-260260, image recognition including an optical flow unit that detects a moving state of an object and a moving object state detecting unit that detects a state of the moving object from a detection result detected from a stereo measurement unit that detects the distance of the object. An apparatus is disclosed. Patent Document 2 discloses that high-accuracy three-dimensional measurement is realized even in a wide range by searching for high-precision parallax using images taken from a plurality of different viewpoints.

Japanese Patent Laid-Open No. 10-222665 Japanese Unexamined Patent Publication No. 2016-65744

In Patent Document 1, it is possible to propose a method of monitoring with a separate sensor or another additional camera for the blind spot problem that cannot be measured in stereo. However, in this case, integration with another sensor and an increase in additional cost are necessary. Moreover, since the information of another sensor and another additional camera does not include three-dimensional information, it is not easy to maintain monitoring accuracy. Also, changing the camera setting value for the shooting direction and zoom of the camera based on the detection result is not disclosed.

Further, in Patent Document 2, although the measurement accuracy is improved with respect to the three-dimensional measurement in a certain range, it is not easy to dynamically adjust the measurement range according to the danger or abnormality that has occurred in the field.

Therefore, an object of the present invention is to provide a highly reliable stereo camera device that takes into account the blind spot area.

In order to solve the above problems, in the stereo camera device according to the present invention, a plurality of actual video input units that capture an image of an object or a region, and a camera control unit that changes a setting value at the time of capturing the actual video input unit, , An image acquisition unit that converts an electronic signal input from the real video input unit, a monitoring unit that detects abnormality or danger based on the image information acquired by the image acquisition unit, and a detection by the monitoring unit Based on the obtained information, a stereo control unit that calculates a control value when the real video input unit is made stereo, and from the real video input unit to the object or region based on the information acquired by the image acquisition unit A distance calculation unit that calculates a distance of the camera, wherein the camera control unit changes a setting value at the time of shooting of the real image input unit based on a control value calculated by the stereo control unit. Characterized in that it.

According to the present invention, it is possible to provide a highly reliable stereo camera device in consideration of the blind spot area.

It is a block diagram which shows the system configuration | structure of one Embodiment by this invention. It is a figure which shows the structural example of a camera control part. It is a figure which shows the structural example of a stereo control part. It is a figure which shows the structural example of a stereo control part. It is a figure which shows the structural example of a distance calculation part. It is a figure which shows the structural example of a distance calculation part. It is a figure which shows 2nd embodiment by this invention. It is a figure which shows 3rd embodiment by this invention. It is a figure which shows 4th embodiment by this invention. It is a figure which shows 5th embodiment by this invention. It is a figure which shows 6th embodiment by this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following are only examples, and are not intended to limit the embodiments of the present invention to the following specific contents.

Here, a stereo camera having two cameras will be described as an example. FIG. 1 is a diagram illustrating a configuration of a stereo camera device according to Embodiment 1 of the present invention.

The real image input unit (stereo camera) 10 basically includes an imaging unit and a lens. The stereo camera receives external images from the left and right camera lenses. The imaging unit is a mechanism including an imaging element such as a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device) which is an image sensor. The lens is a zoomable lens. By zooming the lens, it is possible to image not only the near area but also the telephoto area. In the first embodiment, the scene to be photographed is not specified, but if it is a stereo camera for the monitoring function, a moving object, a surveillance area, or the like can be considered as a photographing target. In the case of an in-vehicle stereo camera, a road or a car ahead can be considered as a shooting scene. The shooting can be performed even in a complicated shooting scene or a simple shooting scene, and the shot video is input. The camera of the first embodiment may be a camera that can change the focal length, the rotation angle, the distance between the cameras, and the like.

The image acquisition unit 11 is a unit that converts an electronic signal input from the real video input unit 10 into an image. Here, there are various image formats, and the image format is determined by the system such as BMP, JPEG, or PNG.

The camera control unit 12 controls the actual video input unit 10. Camera parameters such as camera angle of view and orientation, and distance between cameras can be adjusted. FIG. 2 is a diagram illustrating a configuration example of the camera control unit 12. The camera control unit 12 includes a zoom control unit 21, a tilt control unit 22, a pan control unit 23, and a position control unit 24. The tilt control unit 22 controls the angle of view of the camera and the horizontal direction of the camera, and pans. The control unit 23 controls the angle of view of the camera and the vertical direction of the camera. The zoom control unit 21 controls the lens mechanism, and changes the focal length by moving the lens to change the angle of view that is the imaging range of the camera. The position control unit 24 adjusts the distance between the cameras, and adjusts what is called a baseline in the case of a stereo camera. The tilt control unit 22 and the pan control unit 23 can change the posture while rotating the camera. The above adjustment can be performed independently or simultaneously. This adjustment can be adjusted according to the system specifications and the needs of the scene to be photographed.

Next, returning to FIG. 1, the first embodiment will be described. In the first embodiment, the real video input unit 10, the image acquisition unit 11, and the camera control unit 12 described above are provided in each of the right and left cameras. That is, as shown in FIG. 1, it is comprised by the right demonstration input part, the left demonstration input part, the right image acquisition part, the left image acquisition part, the right camera control part, and the left camera control part.

The monitoring unit 13 uses the image information input from the image acquisition unit 11 to monitor and detect abnormalities and dangers. The abnormalities and dangers here vary depending on the shooting scene. When Example 1 is a vehicle-mounted camera, it corresponds to a time when a person or a vehicle suddenly appears from the front of the operation or a suspicious dangerous object enters. Various methods have been studied for intruder detection. For example, in the method “Matsuyama et al., Background difference robust against lighting changes”, a background model can be reproduced and an intruder can be detected by a monitoring scene. In recent years, a method of detecting a specific person or an abnormal thing from an image by machine learning has been proposed. As a highly versatile method, a combination of SIFT (Scale-Invant Feature Transform), HOG (Histograms of Oriented Gradients) and discriminator Adaboost, SVM (supported vector machine) is used to detect an abnormal combination. There is. Recently, the development of Deep Learning technology and application to abnormality detection have begun. Using the method as described above, it is possible to detect and warn of danger and abnormality in the image. The monitoring unit 13 can also detect an abnormality such as an intrusion from the laser sensor in addition to the method of determining by an image.

The stereo control unit 14 receives the alarm output from the monitoring unit 13, and calculates a control value for the stereo conversion of the camera based on the detected abnormality and danger information. Based on the calculated control value, the camera control unit 12 adjusts the parameters of the right camera and the left camera. This makes the camera stereo.

FIG. 3 is a diagram illustrating a configuration example of the stereo control unit 14. The alarm observation unit 31 determines whether there is an alarm 30 from the monitoring unit 13. When the alarm 30 is detected, the alarm information is output from the alarm observation unit 31 to the alarm analysis unit 32. Here, for example, distance information detected by a laser sensor used in the monitoring unit 13 can be considered as alarm information. The alarm analysis unit 32 specifies a camera to be controlled based on the alarm information. For example, if detection is received from the right camera, it is determined that it is necessary to further measure both cameras in stereo for the observation area of the right camera, and it is possible to decide to control the left camera. . Then, the current camera setting unit 33 acquires the current setting values of the left and right cameras. The installation value means a parameter value of the zoom value, the rotation value, and the inter-camera distance of the camera. The control calculation unit 34 calculates a control value 20 for stereoization based on the camera installation value acquired by the camera current state unit 33 and the result determined by the alarm analysis unit 32, and based on the calculated control value 20, The left camera control unit 13 controls the camera to make it stereo. This operation makes it possible to intelligently control the stereo camera, so that the angle and direction of the stereo camera can be changed separately, and a stereo camera device that can be stereoified dynamically according to abnormalities and dangers. Can be provided. In addition, when two different cameras are made stereo, it is possible to make maximum use of camera resources. Furthermore, if necessary, it can be expected that the cost of the camera will be reduced by making it stereo.

FIG. 4 is a diagram illustrating another configuration example of the stereo control unit 14. Here, the alarm is not based on the information detected by the sensor. This is an example in which an alarm is detected using an image and a control value 20 is estimated. As the alarm information, for example, the position on the image obtained from the right or left camera can be considered. The alarm observation unit is the same as in FIG. The image analysis unit 40 receives the left and right camera images 41 as input and analyzes the danger information on the images. The image analysis unit 40 can analyze a region range where the danger has occurred, a place where the danger has occurred, and the like. For example, when a danger is detected from the image of the right camera, the left camera is controlled to make the left and right cameras stereo. The control estimation unit 42 changes the parameters of the left camera while comparing the images of the left and right cameras, and estimates the control value until it can be made stereo. Then, the estimated control value is output.

Returning to FIG. 1, the first embodiment will be described. When an abnormality or danger occurs in either area of the left and right cameras, the camera control unit 12 quickly controls the camera based on the control value obtained when the stereo control unit 14 acquires the stereo image, and the stereo image is generated. By doing so, it is possible to acquire the abnormal or dangerous state in detail as the left and right camera images and camera information. The distance calculation unit 15 calculates the distance using the acquired left and right camera images and camera information. FIG. 5 is a diagram illustrating a configuration example of the distance calculation unit. The camera calibration unit estimates the current camera parameters from the camera installation value 50. Camera parameters are camera information indicating the focal length, orientation, etc. of the camera, and can be broadly divided into two parameters: internal parameters and external parameters.

K is an internal parameter matrix, f is the focal length, a is the aspect ratio, s is the skew, and (vc, uc) is the center coordinate of the image coordinates. D is an external parameter matrix, (r11, r12, r13, r21, r22, r23, r31, r32, r33) indicates the direction of the camera, and (tx, ty, tz) indicates the world coordinates of the camera installation position. Show. Using these two parameter matrices K, D and constant λ, the image coordinates (u, v) and world coordinates (X, Y, Z) are

These are related by the relational expression. Note that the camera direction (r11, r12,..., R33) indicating the external parameter is represented by three parameters of pan θ, tilt φ, and roll ψ, which are camera installation angles, when defined by Euler angles. Therefore, the number of camera parameters necessary for associating the image coordinates with the world coordinates is 11 which is a total of 5 internal parameters and 6 external parameters. The distortion correction unit 54 corrects the distortion of the camera image using the camera parameter. The parallel processing unit processes the parallelization of the left and right cameras. In other words, the 3D measurement value of the measurement object is

It is possible to calculate as follows. (Xl, yl) and (xr, yr) are pixel values on the left and right camera images, and yl = yr = y after the parallelization processing. f is the focal length, and B is the distance between the baseline and the camera. d is the same three-dimensional and is the difference between the images projected on the left and right cameras. The difference is called parallax in three dimensions. The relationship between world coordinates and image coordinates is

become that way.

FIG. 6 is a diagram illustrating another configuration example of the distance calculation unit 15. The camera automatic calibration unit automatically estimates camera parameters using the input image. Several estimation methods have been developed. One of them is the automatic calibration method described in “Development of self-calibration technology for variable-parameter stereo camera, Li, 22nd Image Sensing Symposium, IS1-04”. After estimating the parameters, the distance can be obtained by the same principle as in FIG.

According to the first embodiment described above, the stereo camera parameters can be intelligently adjusted. In other words, it is possible to detect danger and abnormality by quickly controlling the camera and making it stereo, and to correctly measure the current situation. By using such a stereo camera device, safer and safer control can be performed. For example, when one camera monitors a nearby area and the other camera monitors a telephoto area, if an abnormality or danger occurs in either area, the camera is quickly controlled to make it stereo. It is possible to measure abnormalities and dangers.

An embodiment of the stereo camera device proposed in Embodiment 1 will be described with reference to FIG. In the stereo camera mounted on the conventional automobile 70, the measurement ranges of the left and right cameras are 71 and 72, and 73 and 74 on the image when measuring the vicinity region. On the other hand, when measuring the telephoto area, the measurement ranges of the left and right cameras are 75 and 76, and 77 and 78 on the image.

Here, when both the left and right cameras are distant, a blind spot area 79 is generated in the vicinity area, and a danger or an abnormality 712 is generated on the screen. In this case, there is a problem that the danger cannot be detected and the distance and the measurement cannot be performed. When actually driving a car, these dangers and abnormalities hinder driving, and it is necessary to measure the blind spot area.

In this embodiment, when the stereo camera device is initially activated, the left and right cameras are measured in separate areas, the near area and the telephoto area. Specifically, the left camera measures the near area, and the right camera measures the telephoto area. Then, when the danger 712 is detected by the left camera, the stereo control is immediately performed, and the right camera is controlled so that the telephoto area can be acquired. Then, the area where the danger has occurred is measured with the left and right cameras in a stereo state. As a result, the blind spot area can be eliminated in the vicinity area of the in-vehicle stereo camera, and the danger can be predicted and measured.

FIG. 8 is used to explain another embodiment of the stereo camera device shown in FIG. In this embodiment, when the stereo camera device is initially activated, the left and right cameras are measured in separate areas, a near area and a telephoto area. Specifically, the left camera measures the vicinity area 71, and the right camera measures the telephoto area 76. The image corresponding to the left camera at that time is 73, and the image corresponding to the right camera is 74. When the right camera detects the danger 80, the stereo control is immediately performed and the left camera is controlled. Then, the left and right cameras are measured in a stereo state in the area where the danger 80 has occurred. As a result, the blind spot area can be eliminated from the telephoto area of the in-vehicle stereo camera, and the danger can be predicted and measured.

Also, in this embodiment, when the danger 80 occurs, the dangerous area is a long distance. For this reason, the image region 73 of the camera 71 has a region 75 that is assumed to be distant. Therefore, virtual stereo processing can be performed using the region 75 and the region 74. As a result, stereo three-dimensional measurement can be easily performed without causing camera control. However, in this case, it is predicted that the measurement resolution will be reduced, so it is desirable to apply a super-resolution technique or the like.

Next, Example 4 will be described with reference to FIG. In the fourth embodiment, the functions of the configurations 10 to 12 and 15 shown in FIG. The fourth embodiment includes a monitoring unit 90 that monitors danger and abnormality, and a stereo control unit 91 that instructs the camera to perform stereo control in accordance with the danger and abnormality risk information acquired by the monitoring unit 90. The left and right camera control units 12 are connected to the actual video input unit 10, and the camera control unit 12 adjusts the camera according to the control information of the stereo control unit 91. The image acquisition unit 11 converts the electronic signal input from the real video input unit 10 into an image.

In this embodiment, the monitoring unit 90 and the stereo control unit 91 are connected to the camera control unit 12, the actual video input unit 10, and the image acquisition unit 11 via the Internet. Accordingly, for example, the monitoring unit 90 detects danger or abnormality, and an instruction to stereo-control the camera of the stereo control unit 91 according to the danger information acts on the camera control unit 12 via the Internet line to perform camera control. . The distance calculation unit 15 calculates the distance using the acquired left and right camera images and camera information. Since the distance calculation unit 15 is connected to the real video input unit 10 and the image acquisition unit 11 via the Internet line, the distance calculation unit 15 can acquire camera images and camera information necessary for distance calculation via the Internet.

By connecting each configuration of the stereo camera device via the Internet line in this way, the camera can be controlled at high speed even if each configuration is remotely located, and the camera can be made stereo. Here, in the present embodiment, each component is connected to the Internet as described above, but the Internet connection location may be changed as appropriate.

Next, Example 5 will be described with reference to FIG. In the fifth embodiment, a place where a surveillance camera is installed such as a railway platform is assumed. The stereo camera 100 described in the present embodiment monitors either the near-field or the far-field so that the left and right cameras are different in the vicinity and the distance as proposed in the second and third embodiments. is doing. Here, for example, when a danger is detected such that the child 101 goes outside the yellow line of the railroad platform in the vicinity area, the stereo camera 100 can be made stereo to the vicinity area and monitored, Risk can be prevented.

Further, in the case of crowded 102 where many people are gathering at the railroad platform, it is possible to control the imaging area of the stereo camera 100 by making it stereo, and to grasp the details of the crowded 102. . As a result, it is possible to further grasp whether the traffic is simply crowded or an accident occurs.

Also, if there is a person in a wheelchair at the railroad platform 103, whether the wheelchair user is in trouble can be determined by making the stereo camera 100 stereo with respect to the far field so as to observe the far field. Judgment can be made. Thereby, it is possible to provide services to wheelchair users.

As described above, by appropriately performing stereo processing on one stereo camera as in the present embodiment, it is possible to take an image of a place that seems necessary, and as a result, it is possible to monitor a wide range. The stereo camera device according to the present invention can perform shooting and measurement for multiple purposes, and can provide a more secure and safe service.

Next, Example 6 will be described with reference to FIG. The sixth embodiment is an embodiment in which a stereo camera device using a commercially available PTZ camera is mounted. The PTZ camera is a camera that can control pan, tilt, and zoom of the camera. Each camera's specs and control methods vary from manufacturer to manufacturer. In the present embodiment, two or more PTZ cameras 110 are connected to the camera control unit 113 and the image acquisition unit 114 via the network 112, a cable, and the like to perform camera control and distance measurement. The camera control unit 113 is a unit that controls the PTZ camera 110 and the baseline stage 111. The image acquisition unit 114 is a unit that acquires an image from the PTZ camera via the network 112. The monitoring unit 116 is a unit that monitors abnormalities and dangers on the image using the camera image acquired by the image acquisition unit 114. The stereo control unit 115 instructs the camera control unit 113 to convert the camera into a stereo based on the danger or abnormality detected by the monitoring unit 116. Then, the camera control unit 113 converts the PTZ camera 110 into a stereo based on an instruction from the stereo control unit 115. The distance calculation unit 15 calculates the distance using the camera image and camera information acquired by the PTZ camera, as in the first embodiment.

This embodiment makes it possible to intelligently adjust the parameters of a commercially available PTZ camera, detect dangers and abnormalities, and quickly measure the current state of the stereo image to correctly measure the current situation. By using such a stereo camera device, safer and safer control can be performed. For example, when one of the PTZ cameras monitors a nearby area and the other PTZ camera monitors a far-field area, if an abnormality or danger occurs in either area, the camera is quickly controlled. By making them stereo, abnormalities and dangers can be measured in detail. In this embodiment, a commercially available PTZ camera can be used, the control of the camera can be further simplified, and the development cost can be reduced. Further, when connecting the PTZ camera to each configuration, the measurement range of the camera can be further expanded by using the network 112, and can be adapted to various environments.

Here, in the present embodiment, each component is connected to the network as described above, but the network connection location may be changed as necessary.

10: Real video input unit 11: Image acquisition unit 12: Camera control unit 13: Monitoring unit 14: Stereo control unit 15: Distance calculation unit 20: Control value 21: Zoom control unit 22: Tilt control unit 23: Control pan unit 24 : Pan control unit 30: Alarm 31: Alarm observation unit 32: Alarm analysis unit 33: Camera current status acquisition unit 34: Control calculation unit 35: Right camera installation unit 36: Left camera installation unit 40: Image analysis unit 41: Left and right camera Image 42: Control estimation unit 50: Camera set value 51: Camera calibration unit 52: Camera parameter 53: Image 54: Distortion correction unit 55: Parallel processing unit 56: Distance calculation unit 60: Image 61: Camera automatic calibration unit 62 : Camera parameter 70:

Cars

71, 72, 75, 76:

Measurement areas

73, 74, 77, 78: Acquired image 79: Blind spot area 712: Danger and abnormality 8 : Danger and abnormality 90: Camera position 91: Monitoring unit 92: Stereo control unit 100: Stereo camera 101: Children 102: People group 103: Wheelchair 110: PTZ camera 111: Baseline stage 112: Network 113: Camera control unit 114: Image acquisition unit 115: Stereo control unit 116: Monitoring unit

Claims

A plurality of real video input sections for photographing an object or an area;
A camera control unit for changing a setting value at the time of shooting of the real video input unit;
An image acquisition unit that converts an electronic signal input from the real video input unit;
A monitoring unit for detecting abnormalities and dangers based on the image information acquired by the image acquisition unit;
Based on the information detected by the monitoring unit, a stereo control unit that calculates a control value when the real video input unit is made stereo;
A distance calculation unit that calculates a distance from the actual video input unit to the object or region based on the information acquired by the image acquisition unit;
The said camera control part changes the setting value at the time of imaging | photography of the said real image input part based on the control value which the said stereo control part calculated, The stereo camera apparatus characterized by the above-mentioned.
The stereo camera device according to claim 1,
The said camera control part is a stereo camera apparatus which can change either the focal distance of the said real image input part, a rotation angle, the distance between cameras, or a baseline, or several.
The stereo camera device according to claim 1 or 2,
A stereo camera device characterized in that at least one of the plurality of real video input units is shooting a region farther than a region near another real video input unit.
The stereo camera device according to claim 3,
A stereo camera device characterized in that an actual video input unit that captures the distant area is stereo-ized in the vicinity of the actual video input unit that captures the adjacent area.
The stereo camera device according to claim 3,
A stereo camera device characterized in that a real image input unit that captures the near area is stereo-ized in the distant area with an actual image input unit that captures the distant area.
A stereo camera device according to any one of claims 1 to 5,
The monitoring unit and the stereo control unit are connected to the actual video input unit and the image acquisition unit via the Internet, respectively.
The stereo camera device according to any one of claims 1 to 6,
A stereo camera device, wherein the real image input unit is a PTZ camera.