JP2025082988A - Control unit, control method, and program - Google Patents

Abstract

To appropriately change the range of a cut-out image even when a plurality of partial images is cut out from a picked-up image.SOLUTION: When processing is performed which is to change at least one of the position and the size of a first partial image to be cut out from a picked-up image picked up by imaging means, determination means determines whether to change the angle of view of the picked-up image. When the determination means determines to change the angle of view of the picked-up image, angle of view control means performs control to change the angle of view of the picked-up image to include the area of the first partial image, the position or the size of which is changed through the processing. Cut-out control means performs control to change at least one of the position and the size of a second partial image, which is cut out from the picked-up image and which is different from the first partial image, relative to the angle of view of the picked-up image after the change.SELECTED DRAWING: Figure 3

Description

The present invention relates to control technology for captured images.

In recent years, pan-tilt-zoom cameras (hereinafter referred to as PTZ cameras), which can remotely control the camera's imaging direction in the pan and tilt directions, have been used in video production sites and as security cameras. When producing a video, an external control device called a controller is connected to the PTZ camera via a cable, and the PTZ camera's pan, tilt, and zoom operations are performed remotely by operating a stick on the controller, changing the camera's imaging direction to capture an image of a subject.

Many cameras are also in use that have a crop function that cuts out and outputs any range from a captured image. With cameras that support the crop function, the user can register any range from the captured image as a crop range (partial image). The camera cuts out and resizes the partial image of the specified range from the captured image, and outputs it to an external device via a video output terminal such as HDMI (registered trademark) or SDI (Serial Digital Interface), or via a network.

JP 2016-100636 A JP 2020-122883 A

When changing the range of the crop area in the crop function, the user can only specify it within the range of the captured image. Therefore, if the user wants to change at least one of the position and size of the crop area beyond the image capture area, the user must zoom out once to expand the image capture area, which makes the process complicated. In addition, changing the zoom shifts the angle of view of other crop areas that have already been set, and the user must start over.

Patent Document 1 proposes a technique for adjusting the zoom range so that the imaging range includes a region of interest that exists on the boundary of a specified zoom range in a UI (User Interface) that controls the zoom. However, this technique does not take into account the case where there are multiple regions of interest.

Patent document 2 discloses a technology that changes the crop position of the digital zoom in response to pan/tilt drive. However, this technology is based on the premise that the range to be cropped is always within the range of the captured image, and does not drive the pan/tilt zoom so that the crop position fits within the range.

The present invention aims to make it possible to appropriately change the range of an extracted image even when multiple partial images are extracted from a captured image.

In order to solve the above problem, the control device of the present invention has the following configuration. That is, when a process of changing at least one of the position and size of a first partial image cut out from a captured image captured by an imaging means is performed, the control device has a determination means for determining whether to change the angle of view of the captured image, an angle of view control means for controlling the change of the angle of view of the captured image so as to include the area of the first partial image after the change by the process when the determination means determines that the angle of view of the captured image is to be changed, and a cutout control means for controlling the change of at least one of the position and size of a second partial image cut out from the captured image, the second partial image being different from the first partial image, relative to the angle of view of the captured image after the change when the angle of view of the captured image is changed by the angle of view control means.

According to the present invention, even when multiple partial images are cut out from a captured image, it is possible to appropriately change the range of the cut-out image.

1 is a system configuration diagram according to a first embodiment of the present invention. 1 is an external view of an imaging device according to a first embodiment of the present invention. 1 is a block diagram of an imaging apparatus according to a first embodiment of the present invention; FIG. 4 is a diagram showing a coordinate system of a cropping area according to the first embodiment of the present invention. FIG. 4 is a diagram illustrating an example of a UI according to the first embodiment of the present invention. FIG. 6 is a diagram showing a processing flow when changing a cropping range according to the first embodiment of the present invention. FIG. 11 is a diagram showing a process flow for determining a PTZ position and a crop range 1 position according to the first embodiment of the present invention. 5A to 5C are diagrams showing changes in a captured image and each cropped image according to the first embodiment of the present invention. FIG. 11 is a diagram showing a processing flow when changing a cropping range according to a second embodiment of the present invention. FIG. 11 is a diagram illustrating an example of a communication sequence according to the second embodiment of the present invention. FIG. 11 is a diagram showing a communication packet structure according to a second embodiment of the present invention. 13A to 13C are diagrams showing changes in each cropped image and a full-screen image according to the third embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a hardware configuration according to each embodiment.

The following describes in detail an embodiment of the present invention with reference to the attached drawings. Note that the embodiment described below is one example of a means for realizing the present invention. The present invention should be modified or changed as appropriate depending on the configuration of the device to which the present invention is applied and various conditions, and the present invention is not necessarily limited to the following embodiment. Furthermore, not all of the combinations of configurations described in the embodiments are necessarily essential to the solution of the present invention. Furthermore, the term "image" below does not necessarily mean a moving image, but may also mean a still image.

In the following description, an embodiment will be described in which the position and size of the captured image and crop range are changed. At least one of the position and size of the captured image and crop range may be changed depending on the situation. In other words, depending on the situation, it is not necessary to change both the position and size of the crop range. Also, depending on the situation, it is not necessary to change both the position and size of the captured image. In the following description, when the term "position and size" is used, it is intended to mean "at least one of the position and size."

First Embodiment
A first embodiment of the present invention will be described with reference to Fig. 1 to Fig. 8. Note that the imaging device 100 will be described as having a function of cutting out (cropping) a partial image from a captured image and outputting the partial image to the outside. That is, an example will be described in which the imaging device 100 also functions as a control device for the partial image. However, a system configuration in which a device external to the imaging device 100, such as a client device 200 or a controller 400, has the function of the control device in each embodiment may be used.

Figure 1 is a diagram showing the system configuration in this embodiment. The system in this embodiment has an image capture device (control device) 100, a client device 200, a network 300, a camera controller 400, and a display 500.

The imaging device 100 includes a video output terminal 101, a control communication terminal 102, and a network I/F 103. The video output terminal 101 is a terminal that outputs video captured by the imaging device 100. An example of a video output terminal is a terminal conforming to standards such as SDI or HDMI (registered trademark), but the video may also be packetized using IP (Internet Protocol) and output from an Ethernet (registered trademark) terminal.

The control communication terminal 102 is a communication terminal that receives and responds to control signals and control commands from the camera controller 400. Examples of the control communication terminal include a contact terminal that simply transmits a high/low signal, a serial communication standard such as RS-232C, and a communication standard such as Ethernet (registered trademark). The network I/F 103 is a terminal for communicating with the client device 200 via the network 300. Examples of the network I/F 103 include an Ethernet (registered trademark) terminal and a Wi-Fi (registered trademark) transmission/reception module.

The client device 200 is an information processing device such as a personal computer equipped with a display unit. The client device 200 displays the image generated by the imaging device 100 and a UI (User Interface) based on information received from the imaging device 100 via the network 300. The client device 200 also accepts user operations via the displayed UI and transmits various control instructions to the imaging device 100 based on the operations via the network 300.

The imaging device 100 and the client device 200 are connected to each other via a network 300. The network 300 is realized by a plurality of routers, switches, cables, etc. that comply with a communication standard such as Ethernet (registered trademark). Note that the network 300 may also be realized by the Internet or a wired LAN (Local Area Network), etc.

The camera controller 400 is a controller equipped with a stick and control buttons. When the stick is tilted to the left, an instruction to pan to the left is sent as a control command via the control communication terminal 102 of the imaging device 100, and when the stick is tilted up, an instruction to tilt up is sent as a control command. At that time, the pan or tilt movement speed is determined according to the angle at which the stick is tilted, and a control command specifying the movement speed is sent simultaneously. In addition, the zoom of the imaging device 100 can be changed by twisting the stick. When the stick is twisted to the right, a control command is sent to the imaging device 100 to instruct zooming in, and when the stick is twisted to the left, a control command is sent to the imaging device 100 to instruct zooming out. The stick may be used to change the position and size of a crop range (range of a partial image) to be described later. For example, when the stick is tilted to the left, an instruction to move the position of the crop range to the left is sent to the imaging device 100. When the stick is tilted up, an instruction to move the position of the crop range up is sent to the imaging device 100. When the stick is twisted to the right, an instruction to reduce the crop range is sent to the imaging device 100. Twisting it counterclockwise sends an instruction to the imaging device 100 to enlarge the crop range. The control buttons are used to change the internal settings of the camera controller 400 and to control things other than the PTZ of the camera 100. For example, they can be used to select which of the multiple crops described below is to be output to the video output terminal 101. They are also used to set whether to control the PTZ of the camera 100 or change the position and size of the crop range by operating the stick described above.

The display 500 receives the output image from the camera and displays it to the user.

The imaging device 100 according to this embodiment will be described with reference to Figs. 2 and 3. Fig. 2 is an example of an external view of the imaging device 100 according to this embodiment, with Fig. 2(a) being a view from above and Fig. 2(b) being a view from the side. Fig. 3 is an example of a functional block of the imaging device 100 according to this embodiment.

The direction in which the optical axis of the lens 104 faces is the imaging direction of the imaging device 100, and the light beam that passes through the lens 104 forms an image on the image sensor of the imaging section 110 of the imaging device 100. The lens driving section 105 is composed of a drive system that drives the lens 104, and changes the focal length of the lens 104. The lens driving section 105 is controlled by the pan-tilt-zoom control section 113.

The pan drive unit 106 is composed of a mechanical drive system that performs panning and a motor that is a drive source, and drives to control rotational drive for rotating the imaging direction of the imaging device 100 in the pan direction 108. The pan drive unit 106 is controlled by the pan-tilt-zoom control unit 113.

The tilt drive unit 107 is composed of a mechanism drive that performs tilt operation and a motor that is a drive source, and drives to control rotational drive for rotating the imaging direction of the imaging device 100 in the tilt direction 109. The tilt drive unit 107 is controlled by the pan-tilt-zoom control unit 113.

The imaging unit 110 is composed of an imaging element (not shown) such as a CCD (charge coupled device) sensor or a CMOS (complementary metal oxide semiconductor) sensor. The imaging unit 110 photoelectrically converts the subject image formed through the lens 104 to generate an electrical signal. The image processing unit 111 performs image processing such as converting the electrical signal photoelectrically converted in the imaging unit 110 into a digital signal and compression encoding processing to generate image data. Based on instructions from the system control unit 112, the image processing unit 111 also cuts out a predetermined range of pixels from the image, performs enlargement or reduction processing, and generates a partial image.

The pan-tilt-zoom control unit 113 controls the pan drive unit 106, the tilt drive unit 107, and the lens drive unit 106 based on instructions transmitted from the system control unit 112, thereby controlling the pan, tilt, and zoom of the imaging device 100.

The memory unit 114 stores (holds) settings such as image quality setting parameters, pan/tilt zoom position, and crop range position and size.

The video output unit 115 converts the video generated by the image processing unit 111 into a predetermined video format and outputs it via the video output terminal 101. Furthermore, if the video to be output is a cropped video, and is a vertical crop, as described below, the video output unit 115 rotates the video by 90 degrees before outputting it.

The control communication unit 116 transmits data received from the camera controller 400 via the control communication terminal 102 to the system control unit 112, and performs camera control such as pan-tilt-zoom.

The network communication unit 117 communicates with the client device 200 via the network I/F 103. For example, the network communication unit 117 transmits image data of an image captured by the imaging device 100 to the client device 200 via the network 300. The network communication unit 117 also receives control commands, which are commands for controlling the imaging device 100, transmitted from the client device 200, and transmits them to the system control unit 112.

The image analysis unit 118 performs image analysis on the image generated by the image processing unit 111 based on instructions from the system control unit 112, and transmits the results to the system control unit 112. Examples of image analysis include human body detection, face detection, and moving object detection. For example, a human body is detected in a captured image, and its position is transmitted to the system control unit 112. However, face detection or moving object detection may be used depending on the usage scene.

The system control unit 112 is equipped with a CPU (Central Processing Unit) and controls the entire imaging device 100 by processing executed by the CPU, and performs the following processing, for example. That is, the system control unit 112 analyzes a control command for controlling the imaging device 100 transmitted from the client device 200, and performs processing according to the control command. The system control unit 112 also instructs the pan-tilt-zoom control unit 113 to perform pan-tilt-zoom operation. It also instructs the image processing unit 111 to set image quality adjustment when generating image data, generate partial images, and the like. The system control unit 112 also performs the following processing, for example, according to the image analysis result from the image analysis unit 118. That is, the system control unit 112 receives the position of the human body detected by the image analysis unit 118, determines the imaging direction of the imaging device 100 according to the position, and performs tracking processing of the human body by issuing a pan-tilt-zoom drive instruction to the pan-tilt-zoom control unit 113. Alternatively, the position of the human body detected by the image analysis unit 118 is received, and the position and size of the crop range are determined according to that position, and a crop instruction is issued to the image processing unit 111, thereby moving the crop range so as to track the human body.

In this embodiment, the imaging range is determined by the pan, tilt, and zoom values (in other words, the PTZ position) of the imaging device 100, and the pan and tilt values are expressed by the coordinate system shown in FIG. 2.

The pan value is the angle of the imaging direction (optical axis) in the pan direction 108 of the imaging device 100 when the center of the two drive ends of the pan drive unit 106 is set to 0° as shown in FIG. 2(a). The pan value can take a value in the range of -180° to +180°, with the clockwise direction being positive when viewing the imaging device 100 from above.

The tilt value is the angle of the imaging direction (optical axis) in the tilt direction 109 of the imaging device 100, when the imaging direction of the imaging device 100 is oriented in a direction parallel to the installation surface of the imaging device 100 as shown in FIG. 2(b) is taken as 0°. The tilt value can take a value in the range of -40° to +220°, with the direction pointing upward from the 0° position being positive.

The zoom value of the imaging device 100 when an image is captured with an imaging pressure value of 100 is calculated from the focal length of the lens 104.

Referring to Figure 4, we will explain the coordinate system used to express the position and size of the crop range (partial image) in the crop function (cropping a partial image) of this embodiment.

The captured image 600 corresponds to the entire image captured by the imaging device 100, and its angle of view is determined by the PTZ position of the imaging device 100. In this embodiment, the aspect ratio of the captured image 600 is 16:9, and the number of pixels is 3840 pixels wide and 2160 pixels high. The coordinate system defined on the captured image 600 has the upper left corner as the origin, the X direction to the right, and the Y direction down. In other words, when the coordinates of a pixel on the captured image 600 are (X, Y), the coordinates can range from (0, 0) to (3840, 2160).

The two crop ranges shown in the figure are, for example, crop range 1 601 and crop range 2 602. For example, the aspect ratio of the crop ranges is a horizontal crop of 16:9 and a vertical crop of 9:16. In the example shown in the figure, crop range 601 is a vertical crop and crop range 602 is a horizontal crop.

The position of the crop range is expressed by the center position of the crop range, and for example, the position of the crop range 601 is expressed as (X1, Y1). As an example, it is expressed in a coordinate system with the upper left corner of the captured image as the origin. In other words, X1 is the X coordinate and Y1 is the Y coordinate of such a coordinate system.

The size of the crop area is expressed in terms of the number of pixels in both the vertical and horizontal directions. For example, the width of the crop area 601 is expressed as W1, and the height is expressed as H1.

In the following explanation, the position and size of the nth crop range n will be collectively represented as (Xn, Yn, Wn, Hn). That is, the position and size of crop range 1 601 will be (X1, Y1, W1, H1), and the position and size of crop range 2 602 will be (X2, Y2, W2, H2).

However, the number of cropping ranges does not have to be two, and may be three or more. Also, the aspect ratio of the cropping range is not limited to a specific ratio, and may be any aspect ratio.

An example of a UI for setting a crop according to this embodiment will be described with reference to FIG. 5. This UI is displayed on the display of the client device 200 based on the contents of communication via the network 300 between the imaging device 100 and the client device 200. A user can obtain information about the camera 100 via this UI and set the crop function via an operating device such as a mouse or touch panel. Based on user operations performed on this UI, the client device 200 transmits a setting command to the imaging device 100 via the network 300.

The UI screen 700 is a UI screen related to cropping. The UI screen 700 is displayed, for example, as a web browser screen or a dedicated application window. All of the UI parts described below are displayed on the UI screen 700.

A captured image 600 is displayed on the UI screen 700. In addition, crop ranges 601 and 602 are superimposed on the captured image 600, allowing the user to visually confirm the image currently being captured by the imaging device 100 and the position and size of each crop range. The user can also use the mouse cursor to perform operations such as dragging and dropping the crop ranges 601 and 602 to move the position of the crop ranges or enlarge or reduce their size.

The pan operation slider bar 701 and the pan operation button 702 are UI parts that allow the user to operate the pan of the imaging device 100. By moving the pan operation slider bar 701 by dragging the mouse, for example, an arbitrary pan position can be specified. The pan operation button 702 consists of a left button and a right button, and transmits a command to the imaging device 100 to drive the pan to the left while the left button is pressed, and to drive the pan to the right while the right button is pressed.

The tilt operation slider bar 703 and the tilt operation button 704 are UI parts that allow the user to operate the tilt of the imaging device 100. By moving the tilt operation slider bar 703 by dragging the mouse, for example, an arbitrary tilt position can be specified. The tilt operation button 704 consists of an up button and a down button, and sends a command to the imaging device 100 to tilt up while the up button is pressed, and to tilt down while the down button is pressed.

The zoom operation slider bar 705 and the zoom operation button 706 are UI parts that allow the user to operate the zoom of the imaging device 100. By moving the zoom operation slider bar 705 by dragging the mouse, for example, it is possible to specify any zoom position. The zoom operation button 706 is composed of a Tele button and a Wide button. While the Tele button is pressed, a command is sent to the imaging device 100 to drive the zoom in the Tele direction (zoom in direction), and while the Wide button is pressed, a command is sent to the imaging device 100 to drive the zoom in the Wide direction (zoom out direction).

The crop operation block 710 is a block that brings together UI parts that allow the user to set the position and size of the crop range, and includes the UI parts described below.

The crop selection box 711 sets the crop range to be operated in the crop operation block 710. In this embodiment, this selection box allows the selection of "Crop 1" and "Crop 2." When "Crop 1" is selected, the crop range 601 becomes the target to be operated with the UI parts included in the crop operation block 710; when "Crop 2" is selected, the crop range 602 becomes the target to be operated with the UI parts included in the crop operation block 710.

The crop movement cross button 712 is a button for manipulating the position of the crop range. The crop movement cross button 712 is composed of a left button, a right button, an up button, and a down button. A command is sent to the imaging device 100 to move the crop range leftward while the left button is pressed, rightward while the right button is pressed, upward while the up button is pressed, and downward while the down button is pressed.

The crop zoom operation slider bar 713 and the crop zoom operation button 714 are UI parts that allow the user to operate the size of the crop range. Here, "crop zoom" corresponds to the size of the crop range. Reducing the crop range results in an enlarged display of the range, and increasing the crop range results in a reduced display of the range. Therefore, here, reducing the crop range is expressed as moving the crop zoom in the telephoto direction, and increasing the crop range is expressed as moving the crop zoom in the wide direction. By moving the crop zoom operation slider bar 713 by dragging the mouse, etc., the size of the desired crop range can be specified. The upward direction of the crop zoom operation slider bar 713 is the telephoto direction of the crop zoom, and the downward direction is the wide direction of the crop zoom. The crop zoom operation button 714 consists of a Tele button and a Wide button, and a command to move the crop zoom in the Tele direction while the Tele button is pressed and to move the crop zoom in the wide direction while the Wide button is pressed is sent to the imaging device 100.

The crop movement speed operation slider bar 715 and the crop movement speed operation button 716 are UI parts for specifying the movement of the crop range and the speed of the crop zoom. While the crop movement cross button 712 and the crop zoom operation button 714 are pressed, the movement of the crop range and the movement of the crop zoom are performed at a speed that corresponds to the crop movement speed. The crop movement speed can be specified by moving the crop movement speed operation slider bar 715 by dragging the mouse, etc. The crop movement speed operation button 716 consists of a Fast button and a Slow button, and a command is sent to the imaging device 100 to speed up the crop movement speed while the Fast button is pressed, and to slow down the crop movement speed while the Slow button is pressed.

The crop output setting block 720 is a block that allows the user to select which cropped image to output for each video output, or whether to output a full screen image instead of a cropped image, and is provided with an output selection box 721 for each video output. In this embodiment, there are SDI, stream 1, stream 2, and stream 3 as video output systems. SDI is an image that is output via the video output terminal 101, and streams 1 to 3 are images that are output as network streams via the network I/F 103. In the example of FIG. 5, the settings are set to output crop 1 image to SDI, full screen image to streams 1 and 3, and crop 2 image to stream 2.

The crop angle fixation block 730 is a block for setting whether to fix the crop angle of view, and includes a field angle fixation selection box 731 for each crop. The field angle fixation setting has the options of "always OFF", "when subject is detected", "when outputting", and "always ON". This is a setting for determining whether to correct the position and size of the crop range when the PTZ position of the imaging device 100 is moved. The correction of the crop range when the PTZ position is moved will be explained in detail later. If "always OFF" is selected, the crop range is not subject to correction. If "when subject is detected" is selected, the crop range is subject to correction when a target subject (e.g. a person) is detected within the crop range. If "when outputting" is selected, the crop range is subject to correction when the target crop is output to SDI or any of the video output systems from stream 1 to stream 3. If "always ON" is selected, the crop range is always subject to correction. Whether it is set to "always ON" or "always OFF" is managed by a flag. The memory unit 114 stores the setting value of this flag.

The angle of view priority setting block 740 is a block for setting whether the angle of view is to be prioritized, full screen or crop, and includes an angle of view priority setting radio button 741. If full screen is selected with the angle of view priority setting radio button 741, the process of moving the PTZ accordingly when the position or size of the crop range is changed is not performed. If crop is selected with the angle of view priority setting radio button 741, PTZ movement may be performed when the position or size of the crop range is changed. In addition, when performing PTZ movement, a restriction is imposed that prevents movement to a PTZ position where the crop angle of view cannot be maintained.

The crop auto-tracking setting block 750 is a block that sets the auto-tracking of crops, and includes a crop auto-tracking switch 751 for each crop. A crop with the crop auto-tracking switch 751 turned ON moves the crop range so as to automatically track a subject within the crop range. Detection of a subject within the crop range is performed by the image analysis unit 118, and if a subject is present within the crop range, the position and size of the crop range are automatically changed to track the subject.

The correction process when changing the angle of view of the crop according to this embodiment will be described with reference to Figures 6, 7, and 8. Figure 6 is a flowchart of the process when changing the crop range of the imaging device 100. Figure 7 is a flowchart explaining the process of step S602 in Figure 6 in detail. Figure 8 is a diagram showing the captured image 600 and changes in the output of each partial image. Partial image 801 of crop range 1 is an image that is output by cutting out crop range 601. Partial image 802 of crop range 2 is an image that is output by cutting out crop range 602. Note that, although a change to increase the size of crop range 601 will be described as an example here, the same applies to a change to crop range 602.

In step S601, the imaging device 100 acquires the contents of the position and size change for the crop range 601 through the network communication unit 117. One example of the change is a command to change the size of the crop range 601 sent from the client device 200 based on a user operation using the crop zoom operation slider bar 713 or the crop zoom operation button 714. In this case, the command sent to the system control unit 112 through the network communication unit 117 is analyzed to acquire the contents of the change in the position and size of the crop range 601. In addition, if crop auto-tracking of the crop range 601 is enabled, the crop range may be moved or changed in size based on the auto-tracking result. In this case, the system control unit 112 determines the position and size of the crop range 601 based on the position of the tracking target detected by the image analysis unit 118. FIG. 8A shows the captured image and the state of each crop range at this point. Here, it is assumed that a command to change the crop range 601 to be larger as shown by the dashed line in the figure has been received.

In step S602, the system control unit 112 determines the PTZ position and the position and size of the crop area 601 based on the acquired changes to the position and size of the crop area 601 and the current settings. This determination method will be described in detail later with reference to FIG. 7.

In step S603, based on the determination in step S602, the system control unit 112 changes the position and size of the crop area 601.

In step S604, based on the determination in step S602, the system control unit 112 determines whether or not it is necessary to move the PTZ position of the imaging device 100. If PTZ movement is not necessary, the process ends here. If PTZ movement is necessary, the process proceeds to step S605. As described above, the system control unit 112 functions as a determination unit that determines whether to change the angle of view of a captured image when a process for changing the position and size of a first partial image cut out from the captured image is performed.

In step S605, the system control unit 112 starts moving the angle of view of the captured image to the PTZ position determined in step S602. Thereafter, steps S606 to S610 are repeatedly executed until the PTZ position reaches the target position. In this way, when it is determined that the angle of view of the captured image is to be changed, the system control unit 112 also functions as an angle of view control unit that performs control to change the angle of view of the captured image so as to include the area of the first partial image after the change process.

In step S606, the current PTZ position is obtained. The current PTZ position can be obtained via the pan-tilt-zoom control unit 113.

In step S607, in response to a change in the PTZ position, the system control unit 112 determines whether correction of the crop range 602 is necessary.

As an example, this determination may be made taking into consideration the settings of the output selection box 721 and the fixed angle of view selection box 731. In that case, the following three conditions may be considered as conditions for determining that correction of the crop range 602 is necessary:

The first case is when crop 2 is set to "always on" in the angle of view fixing selection box 731. In this case, it is determined that correction of the crop range 602 is necessary. Whether it is set to "always on" is determined by a flag.

The second case is when crop 2 is set to "when subject is detected" in the fixed angle of view selection box 731, and a specific subject of interest is detected within crop range 2. In this case, it is determined that correction of crop range 602 is necessary.

The third case is when crop 2 is set to "at output" in the fixed angle of view selection box 731, and "crop 2" is set in one of the selection boxes in the output selection box 721. In other words, this is the case when a partial image based on crop 2 is being output externally. In this case, it is determined that correction of the crop range 602 is necessary.

If none of the above conditions are met, it is determined that correction of the crop range 602 is unnecessary, and the process proceeds to step S610. Note that the above three conditions are merely examples, and the crop range may always be corrected without making a judgment based on these conditions. In other words, the position and size of other crop ranges may always be corrected without relying on flags, etc.

If it is determined that correction of the crop range 602 is necessary, proceed to step S608.

In step S608, the system control unit 112 calculates the position and size of crop range 2 that does not substantially change the angle of view of crop range 602. In other words, it calculates the position and size of crop range 2 that will result in a partial image of substantially the same subject even if the angle of view of the captured image changes due to PTZ movement. This calculation method will be described in detail below.

Here, the PTZ position is represented as (P, T, Z), the PTZ position before the start of the PTZ movement, i.e., at the time of step S605, is represented as (Pa, Ta, Za), and the PTZ position at the time of execution of the current step S608 is represented as (Pb, Tb, Zb). However, the zoom value Z is represented as the horizontal angle of view here, with the unit being [deg]. Also, the position and size of the crop range 2 before the start of the PTZ movement, i.e., at the time of step S605, is represented as (X2a, Y2a, W2a, H2a), and the position and size of the corrected crop range 2 is represented as (X2b, Y2b, W2b, H2b). Then, since the number of pixels of the captured image 600 is 3840 x 2160, (X2b, Y2b, W2b, H2b) can be calculated by the following formula (1).

In step S609, the system control unit 112 controls the image processing unit 111 to change the position and size of crop range 2 to the position and size calculated by equation (1). In this way, when the angle of view of the captured image is changed, the system control unit 112 also functions as a cropping control unit that controls to change the position and size of crop range 2 with respect to the changed captured image.

In step S610, it is determined whether the PTZ has reached the target position, and if not, the process proceeds to step S606 again. If the PTZ has reached the target position, the process proceeds to step S611, the PTZ is stopped, and the flow ends.

Figure 8(b) shows the captured image 600 after the series of steps are completed, as well as each crop range and each corresponding partial image. The angle of view of partial image 801 in crop range 1 has been changed widely, and the angle of view (subject) of partial image 802 in crop range 2 has not changed as a result.

The processing content of step S602 will be explained in detail with reference to FIG. 7.

In step S701, the system control unit 112 determines whether the new position and size of the crop range 601 acquired in step S601 will exceed the allowable range of the crop range in the captured image 600. The allowable range of the crop range is the range within the captured image 600 within which the crop range must fall. The allowable range of the crop range can be determined, for example, as the range excluding the outer edge area of a certain number of pixels (let's say K pixels) from the edge of the angle of view of the captured image 600. The position and size (X, Y, W, H) of the crop range must satisfy the following formula (2).

If all of the constraints in equation (2) are satisfied, it can be determined that crop range 1 falls within the crop range tolerance. The crop range tolerance is set because if the crop range is located too close to the edge of the angle of view of the captured image 600, even a small movement of the subject when crop auto tracking is ON will cause the subject to move outside the range of the captured image 600, making tracking impossible. Therefore, by setting the crop range tolerance, it is possible to prevent crop auto tracking from being interrupted.

In step S702, the system control unit 112 determines whether the new crop range 601 falls outside the allowable range of the crop range. If it does not fall outside the allowable range, the system control unit 112 proceeds to step S703, and if it does fall outside the allowable range, the system control unit 112 proceeds to step S704.

In step S703, the system control unit 112 uses the position and size of the new crop range 1 as is, since there is no need to change the PTZ, and ends the flow.

In step S704, the system control unit 112 checks the angle of view priority setting to determine whether to move the PTZ. If crop range priority is set, the process proceeds to step S705 to move the PTZ position. If full screen priority is set, the process proceeds to step S709 to not move the PTZ position. Note that step S704 may be omitted and the process proceeds to step S705.

In step S705, the system control unit 112 performs calculations to determine the PTZ position so that the entire crop range with a fixed angle of view falls within the imaging range. An example of a calculation method for determining this PTZ position is described below. However, in this explanation, crop range 2 is assumed to have a fixed angle of view.

The position and size of the new crop range 601 are (X1a, Y1a, W1a, H1a). The position and size of the crop range 602 at this point are (X2a, Y2a, W2a, H2a). Furthermore, the numbers of pixels that can contain all the crop ranges are Wcrop_a and Hcrop_a, respectively. In this case, the condition for crop range 1 and crop range 2 to be included in the captured image just by driving the PT without changing the zoom is that the following formula (3) is satisfied.

where K is the number of pixels in the outer edge region used to determine the crop range tolerance, as described above. If the inequality in formula (3) holds, there is no need to move the zoom in the wide direction, so the zoom can be fixed and moved to a PT position that includes crop range 601 and crop range 602. If formula (3) does not hold, it is necessary to move the zoom in the wide direction, so the zoom position is calculated.

The current horizontal angle of view of the zoom is Za, and the horizontal angle of view of the zoom after the movement is Zb. The position and size of the new crop range 601 and the current crop range 602 corrected for the zoom movement are (X1b, Y1b, W1b, H1b) and (X2b, Y2b, W2b, H2b), respectively. Here, the following formula (4) must be satisfied for the position and size of the corrected crop range.

Here, the positional relationship between crop range 601 and crop range 602 does not change due to zooming, and both are equally affected by correction, so the following formula (5) holds.

Therefore, once Wcrop_a and Hcrop_a are found, the zoom position can be set to Zb that satisfies both of the following two equations (6).

After calculating the zoom amount as described above, the image can be moved to a PT position at the changed zoom position where both crop range 601 and crop range 602 can be included in the captured image.

In step S706, if Zb exists that satisfies both of the above formulas (6), it is determined that a suitable PTZ position exists, and the process proceeds to step S707. If formula (6) is not satisfied even when the zoom is pulled all the way to the wide-angle end, it is determined that a suitable PTZ position does not exist, and the process proceeds to step S708.

In step S707, the system control unit 112 sets the PTZ position determined in step S705 as the determined position, and proceeds directly to step S710.

In step S708, the system control unit 112 determines that the zoom setting is the wide end, and determines the PT position to be the position that maximizes the crop range 601. The PT position may be determined based on the positional relationship between the crop range 601 and the crop range 602. For the horizontal direction, assume that the X position X1 of the crop range 601 is smaller than the X position X2 of the crop range 602. In this case, the right end of the crop range 2 is set to the right end of the crop range allowable range. If X1 is larger than X2, the pan position is determined so that the left end of the crop range 2 is set to the left end of the crop range allowable range. For the vertical direction, if the Y position Y1 of the crop range 601 is smaller than the Y position Y2 of the crop range 602, the bottom end of the crop range 2 is set to the bottom end of the crop range allowable range. If Y1 is larger than Y2, the tilt position is determined so that the top end of the crop range 2 is set to the top end of the crop range allowable range.

In step S709, the system control unit 112 determines the size of crop range 1 to be the maximum size that fits within the captured image. Since the PTZ position has already been determined, it is sufficient to determine the maximum size that makes the above-mentioned equation (2) upright for the determined PTZ position and the position of crop range 601. With this procedure, as shown in FIG. 8C, it is possible to maximize the angle of view of partial image 801 in crop range 1 without changing the angle of view of partial image 802 in crop range 2.

In step S710, the system control unit 112 corrects the position and size of the crop range 601 in accordance with the movement to the PTZ position determined in the previous steps. This correction can be performed using the above-mentioned formula (1).

The above is the first embodiment. By controlling the cut-out range as described above, it is possible to appropriately change the range of the cut-out image even when multiple partial images are cut out from a captured image.

Second Embodiment
Next, a second embodiment of the present invention will be described with reference to FIGS.

Note that the system configuration diagram, external view of the imaging device, functional block diagram, and coordinate system of the crop range according to this embodiment are the same as those described with reference to Figures 1 to 5 in the first embodiment, and therefore descriptions will be omitted where appropriate. Descriptions of other parts similar to those in the first embodiment will be omitted where appropriate.

In the first embodiment, the image capture device 100 acquires the changes to the position and size of the crop range, and determines the changes to the PTZ position and the corrections to the position and size of the crop range. However, the changes to the PTZ position and the corrections to the position and size of the crop range may be determined on the client device 200 side. In this embodiment, an example will be described in which the changes to the PTZ position and the corrections to the position and size of the crop range are determined on the client device 200 side, and transmitted to the image capture device 100 by command via the network 300.

Fig. 9 is a flowchart of processing on the client device 200 side when changing the position and size of the crop range 601 according to this embodiment, and Fig. 10 is an example of a communication sequence between the imaging device 100 and the client device 200. A series of communications is realized by HTTP (Hypertext Transfer Protocol) requests and responses on a communication path, for example, a wired LAN. The data exchanged here will be described later. In this embodiment, it is assumed that functions equivalent to the system control unit 112 and network communication unit 117 are located inside the client device 200, and the client device 200 acquires the necessary information by communicating with the imaging device 100.

In step S901, the system control unit 112 acquires the changes to the position and size of the crop range 601. These changes are, for example, operations performed using the crop movement cross button 712 or the crop zoom operation slider bar 713.

In step S902, the system control unit 112 determines the PTZ position and the position and size of the crop range 601. This determination method is similar to the content of step S602 described with reference to FIG. 6 in the first embodiment, so a description thereof will be omitted.

In step S903, the system control unit 112 sends a command to the imaging device 100 to specify the position and size of the crop area 601.

In step S904, the system control unit 112 determines whether PTZ movement is necessary based on the determination made in step S902. If PTZ movement is not necessary, this flow ends. If PTZ movement is necessary, the process proceeds to step S905.

In step S905, the system control unit 112 sends a command to the imaging device 100 to instruct it to perform PTZ movement.

In step S906, the system control unit 112 acquires the current PTZ position of the imaging device 100. The PTZ position is acquired by transmitting a PTZ position information request command from the client device 200 to the imaging device 100, and the imaging device 100 transmitting PTZ position information as a response thereto.

In step S907, the system control unit 112 determines whether or not correction of the position and size of the crop area 602 is necessary. The method of this determination is the same as the processing of step S607 described with reference to FIG. 6 in the first embodiment, and therefore will not be described here.

In step S908, the system control unit 112 calculates the position and size of the crop range 602 so that the angle of view does not change. This process is similar to the process of step S608 described with reference to FIG. 6 in the first embodiment, and therefore will not be described.

In step S909, the system control unit 112 causes the network communication unit 117 to send a command to the device 100 to change the position and size of crop area 2.

In step S910, it is determined whether the PTZ has reached the target position. If the PTZ has not reached the target position, the process returns to step S906. If the PTZ has reached the target position, the process ends.

Figure 11 shows an example of the packet structure of data exchanged in the communication sequence described in Figures 9 and 10. Figure 11(a) shows an example of data transmitted from the client device 200 to the imaging device 100, and Figure 11(b) shows an example of data transmitted from the imaging device 100 to the client device 200.

The data transmitted from the client device 200 to the imaging device 100 consists of a header section, a command section, and a parameter section. The header section contains information such as the address of the device that sent the packet, the address of the device that sent the packet, and the size of the packet. The command section represents the function of the imaging device 100 that is controlled by this packet, and the parameter section represents the detailed control content of the command. The number of parameters in the parameter section varies depending on the content of the command. The command available is "PTZ_GET_POS" which requests information on the current PTZ position of the imaging device 100. In addition, "PTZ_SET_POS" which specifies the PTZ position of the imaging device 100 and requests PTZ movement, and "CROP_GET_POS_SIZE" which requests information on the position and size of an arbitrary crop range are also available. Furthermore, "CROP_SET_POS_SIZE" which changes the position and size of an arbitrary crop range can also be used. In this embodiment, the aspect ratio of the crop range is fixed, so "CROP_SET_POS_SIZE" specifies only the width of the crop range.

The data transmitted from the imaging device 100 to the client device 200 is composed of a header section, multiple key sections, and a data section. The header section contains information such as the address of the device that sent the packet, the address of the device that sent the packet, and the size of the packet. The key section is an identifier that indicates the content represented by the data, and the data section contains a value. As keys, "PAN", "TILT", and "ZOOM" that indicate the PTZ position of the imaging device 100 can be used. Also, "CROP1_X", "CROP1_Y", "CROP1_W", "CROP1_H", "CROP2_X", "CROP2_Y", "CROP2_W", and "CROP2_H" that indicate the position and size of each crop range are also used.

Third Embodiment
A third embodiment of the present invention will be described with reference to FIG.

Note that the system configuration diagram, external view of the imaging device, functional block diagram, coordinate system of the crop range, and processing flow when changing the crop range 601 according to this embodiment are the same as those described with reference to Figures 1 to 7 in the first embodiment, and therefore descriptions thereof will be omitted. Descriptions of other parts similar to those in the first embodiment will be omitted as appropriate.

In the first embodiment, if a change in the PTZ position occurs due to a change in the crop range 601, the angle of view of the full-screen image changes. In this embodiment, when the PTZ position is changed and the imaging area after the change includes the imaging area before the change, a process is included in which a cropped image is output as the full-screen image so that the angle of view of the full-screen image does not change.

Figure 12 shows the captured image and each crop range in this embodiment, and the changes between the full screen image and each cropped image. Figure 12(a) shows the image before the crop range 601 is changed, and Figure 12(b) shows the image after the change. The full screen image 800 is a full screen image that is not cropped, and is output to the output system for which "full screen" is specified in the output selection box 721.

As shown by the dashed line in FIG. 12A, the change in size of the crop range 601 causes the crop range 601 to go outside the range of the captured image 600. At this time, the PTZ position of the imaging device 100 is changed according to the flow described with reference to FIG. 6 and FIG. 7 of the first embodiment. Furthermore, by correcting the position and size of the crop range 602, the angle of view of the partial image 801 in the crop range 1 can be widened without changing the angle of view of the partial image 802 in the crop range 2. At this time, the system control unit 112 sets a full-screen image cut-out range 620 within the captured image 600. The position and size of the full-screen image cut-out range 620 can be determined by performing the calculation process performed in step S608 of FIG. 6 on the entire range of the captured image 600 in the same way as the crop range 602. The system control unit 112 generates a full-screen image 800 by having the image processing unit 111 perform cut-out and resizing processes similar to cropping according to the determined full-screen image cut-out range 620. As a result, as shown in FIG. 12(b), the angle of view of the partial image 801 in crop range 1 can be widened without changing the angle of view of the full-screen image 800 before and after changing the PTZ position.

Other Examples
Next, a hardware configuration for realizing each function of the control device of each embodiment will be described with reference to Fig. 13. Note that each component of the camera other than the image sensor and lens in each embodiment may also be realized by the hardware configuration shown in Fig. 13. These pieces of hardware are included in the image capture device 100, the client device 200, or the controller 400.

RAM (RANdom access memory) 1302 temporarily stores computer programs executed by CPU (Central Processing Unit) 1301. RAM 1302 also temporarily stores data (commands and video data) acquired from the outside via communication interface 1304. RAM 1302 also provides a work area used by CPU 1301 when executing various processes. RAM 1302 also functions as, for example, a frame memory or a buffer memory.

The CPU 1301 executes computer programs stored in the RAM 1302. In addition to the CPU, a processor such as a DSP (DigitAl SignAl Processor) or an ASIC (Application Specific Integrated Circuit) may also be used.

HDD (Hard Disk Drive) 1303 stores operating system programs and video data. HDD 1303 also stores computer programs.

Computer programs and data stored in HDD 1303 are loaded into RAM 1302 as appropriate under the control of CPU 1301, and executed by CPU 1301. Other storage media such as flash memory may be used in addition to HDD. Bus 1305 connects each piece of hardware. Each piece of hardware exchanges data via bus 1305. The above is the hardware configuration in each embodiment.

The present invention can also be realized by a process in which one or more processors read and execute a program that realizes one or more functions of the above-mentioned embodiments. The program may be supplied to a system or device having a processor via a network or a storage medium. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions of the above-mentioned embodiments.

The control device in each embodiment may be realized by the hardware shown in FIG. 13, or by software.

Furthermore, the present invention is not limited to the embodiments described above, and various modifications are possible without departing from the spirit of the present invention. For example, combinations of the embodiments and modifications are also included in the disclosure of this specification.

The disclosure of this specification includes the following image processing device, image processing method, and program.

(Item 1)
a determination means for determining whether to change an angle of view of a captured image when a process for changing at least one of a position and a size of a first partial image cut out from a captured image captured by an imaging means is performed;
a view angle control means for performing control to change the view angle of the captured image so as to include an area of the first partial image after the change by the processing when the determination means determines that the view angle of the captured image is to be changed;
When the angle of view of the captured image is changed by the angle of view control means,
a cut-out control means for controlling to change at least one of the position and size of a second partial image cut out from the captured image, the second partial image being different from the first partial image, relative to the angle of view of the captured image after the change.

(Item 2)
2. The control device according to item 1, wherein the determination means determines whether to change the angle of view of the captured image based on a position of the first partial image and a content of the processing.

(Item 3)
3. The control device according to claim 1, wherein the processing is performed in accordance with an operation by a user.

(Item 4)
The control device according to any one of items 1 to 3, characterized in that the processing is performed according to a result of tracking a subject in the captured image.

(Item 5)
The control device according to any one of items 1 to 4, characterized in that the determination of whether to change the angle of view of the captured image is made based on a setting set by a user indicating whether the captured image or the first partial image is to be prioritized.

(Item 6)
storing a flag indicating whether to correct at least one of a position and a size of the captured image for each of the plurality of partial images;
The control device described in any one of items 1 to 5, characterized in that when the angle of view of the captured image is changed by the angle of view control means, the cropping control means determines whether to change at least one of the position and size of the second partial image in the captured image based on the flag in the second partial image.

(Item 7)
The control device described in any one of items 1 to 6, characterized in that when the angle of view of the captured image is changed by the angle of view control means, the cropping control means determines whether to change at least one of the position and size of the second partial image in the captured image based on whether the second partial image is output to the outside.

(Item 8)
The control device described in any one of items 1 to 7, characterized in that when the angle of view of the captured image is changed by the angle of view control means, the cropping control means determines whether to change at least one of the position and size of the second partial image in the captured image based on whether the second partial image contains a specific subject.

(Item 9)
a determination step of determining whether to change an angle of view of a captured image when a process of changing at least one of a position and a size of a first partial image cut out from a captured image captured by an imaging means is performed;
a view angle control step of performing control to change the view angle of the captured image so as to include an area of the first partial image after being changed by the processing, when it is determined in the determination step that the view angle of the captured image is to be changed;
When the angle of view of the captured image is changed by the angle of view control step,
a cut-out control process for controlling a change in at least one of the position and size of a second partial image cut out from the captured image, the second partial image being different from the first partial image, relative to the angle of view of the captured image after the change.

(Item 10)
10. A program for causing a computer to execute the control method according to item 9.

Reference Signs List 100 Imaging device 200 Client device 300 Network 400 Camera controller 111 Image processing unit 112 System control unit 113 Pan-tilt-zoom control unit 117 Network communication unit

Claims (10)

a determination means for determining whether to change an angle of view of a captured image when a process for changing at least one of a position and a size of a first partial image cut out from a captured image captured by an imaging means is performed;
a view angle control means for performing control to change the view angle of the captured image so as to include an area of the first partial image after the change by the processing when the determination means determines that the view angle of the captured image is to be changed;
When the angle of view of the captured image is changed by the angle of view control means,
a cut-out control means for controlling to change at least one of the position and size of a second partial image cut out from the captured image, the second partial image being different from the first partial image, relative to the angle of view of the captured image after the change. The control device according to claim 1 , wherein the determining means determines whether to change the angle of view of the captured image based on the position of the first partial image and the content of the processing. The control device according to claim 1 , wherein the process is performed in accordance with an operation by a user. The control device according to claim 1 , wherein the processing is performed according to a result of tracking the subject in the captured image. The control device according to claim 1 , wherein the determination as to whether to change the angle of view of the captured image is made based on a setting set by a user indicating whether the captured image or the first partial image is to be prioritized. storing a flag indicating whether to correct at least one of a position and a size of the captured image for each of the plurality of partial images;
The control device described in claim 1, characterized in that when the angle of view of the captured image is changed by the angle of view control means, the cropping control means determines whether to change at least one of the position and size of the second partial image in the captured image based on the flag in the second partial image. The control device described in claim 1, characterized in that when the angle of view of the captured image is changed by the angle of view control means, the cropping control means determines whether to change at least one of the position and size of the second partial image in the captured image based on whether the second partial image is output to the outside. The control device described in claim 1, characterized in that when the angle of view of the captured image is changed by the angle of view control means, the cropping control means determines whether to change at least one of the position and size of the second partial image in the captured image based on whether the second partial image contains a specific subject. a determination step of determining whether to change an angle of view of a captured image when a process of changing at least one of a position and a size of a first partial image cut out from a captured image captured by an imaging means is performed;
a view angle control step of performing control to change the view angle of the captured image so as to include an area of the first partial image after being changed by the processing, when it is determined in the determination step that the view angle of the captured image is to be changed;
When the angle of view of the captured image is changed by the angle of view control step,
a cut-out control process for controlling a change in at least one of the position and size of a second partial image cut out from the captured image, the second partial image being different from the first partial image, relative to the angle of view of the captured image after the change. A program for causing a computer to execute the control method described in claim 9.

Images