WO2019171635A1

WO2019171635A1 - Operation input device, operation input method, anc computer-readable recording medium

Info

Publication number: WO2019171635A1
Application number: PCT/JP2018/034490
Authority: WO
Inventors: 夏美鈴木
Original assignee: Ｎｅｃソリューションイノベータ株式会社
Priority date: 2018-03-07
Filing date: 2018-09-18
Publication date: 2019-09-12
Also published as: CN111886567B; JPWO2019171635A1; JP6898021B2; CN111886567A

Abstract

An operation input device 100 comprising a display device 10 for displaying an operation screen, an optical plate 20 for projecting the operation screen in the air to generate a midair projected surface 21, a sensor device 30, and a control device for identifying an operation input. The sensor device 30 outputs sensor data that includes information for identifying the two-dimensional coordinates of an object in a sensing area, and the depth to the object 50. From the sensor data the control device 40 detects the position of the object 50 on the midair projected surface 21, and when a portion of the object is positioned on the sensor-device side of the midair projected surface 21 the control device sets in the sensor data a figure surrounding the portion of the object 50, and uses the depth of the outer edge of the figure to correct the position of the object 50 on the midair projected surface 21.

Description

Operation input device, operation input method, and computer-readable recording medium

The present invention relates to an operation input device and an operation input method that enable an input operation by touching a screen displayed in the air. Furthermore, the present invention relates to a computer-readable recording medium on which a program for realizing these is recorded. About.

In recent years, an operation device has been proposed in which an operation screen is projected in the air, and a user can input an operation by touching the screen projected in the air (hereinafter referred to as “aerial projection plane”). (For example, refer to Patent Document 1). The operation device proposed by Patent Literature 1 includes a display device, an image imaging plate that forms an image of the screen of the display device in the air, a camera, a distance sensor, and a control unit.

Among these, the image imaging plate has a function of collecting the light emitted from the image at a specific position at the same distance on the opposite side as viewed from the image imaging plate to form the same image (for example, , See Patent Document 2). For this reason, when the light emitted from the screen displayed on the display device passes through the image imaging plate, the screen is projected into the air.

In the operation device disclosed in Patent Document 1, the camera captures the aerial projection plane and the user's finger, and inputs the captured image to the control unit. The distance sensor then measures the distance from the user's fingertip and inputs the measured distance to the control unit.

The control unit first calculates the coordinates of the user's finger on the aerial projection plane by substituting the position on the image of the user's fingertip shown in the captured image and the distance measured by the distance sensor into the conversion formula. To do. The conversion formula used at this time is determined in advance from the position coordinates of the aerial projection plane and camera information (position, angle of view, focal length, etc.).

Subsequently, the control unit determines the overlap between the user's finger and the operation icon displayed on the aerial projection plane based on the calculated coordinates, and specifies the input operation by the user based on the determination result. With such a configuration, the user can perform an input operation by touching a screen displayed in the air.

JP 2017-62709 A JP 2012-14194 A

However, the operation device disclosed in Patent Document 1 can accurately detect the coordinates of the contact position of the user's finger on the aerial projection plane when the user's fingertip is positioned behind the operation screen in the air. There is a problem of disappearing.

Specifically, when the user's fingertip is located behind the aerial projection surface in the air, the distance measured by the distance sensor is shorter than the distance at the contact position intended by the user. . As a result, for example, assuming that the distance sensor is installed so that the distance at the center position of the aerial projection plane is the shortest, the coordinates of the user's finger on the aerial projection plane are Will move to the center of the screen.

An example of an object of the present invention is to solve the above-described problem and suppress a decrease in detection accuracy of a touch position due to an erroneous operation of a user when a user performs an input operation by touching a screen displayed in the air. An operation input device, an operation input method, and a computer-readable recording medium are provided.

In order to achieve the above object, an operation input device according to one aspect of the present invention includes:
A display device that displays an operation screen, an optical plate that projects the operation screen into the air to generate an aerial projection surface, and a sensor device for detecting the position of an object that contacts the aerial projection surface in a three-dimensional space And a control device that identifies an operation input performed on the operation screen, and the sensor device includes information for identifying a two-dimensional coordinate of the object in a sensing area, and the sensor device Output sensor data including depth to the object,
The controller is
From the sensor data, detect the position of the object in the aerial projection plane,
Further, when detecting from the sensor data that a part of the object is located on the sensor device side of the aerial projection surface, a figure surrounding the part of the object is set in the sensor data; and Using the depth at the set outer edge of the figure, the position of the object on the aerial projection plane is corrected.
It is characterized by that.

In order to achieve the above object, an operation input method according to one aspect of the present invention includes:
A display device that displays an operation screen, an optical plate that projects the operation screen into the air to generate an aerial projection surface, and a sensor device for detecting the position of an object that contacts the aerial projection surface in a three-dimensional space And a computer for specifying an operation input performed on the operation screen, and information for the sensor device to specify the two-dimensional coordinates of the object in a sensing area, and the sensor device from the sensor device An operation input method for outputting sensor data including a depth to an object,
(A) detecting a position of the object on the aerial projection plane from the sensor data by the computer;
(B) When the computer detects from the sensor data that a part of the object is located on the sensor device side of the aerial projection surface, a figure surrounding the part of the object is displayed in the sensor data. Set, step, and
(C) correcting the position of the object on the aerial projection plane using the depth at the outer edge of the graphic set by the computer,
It is characterized by that.

Furthermore, in order to achieve the above object, a computer-readable recording medium according to one aspect of the present invention is provided.
A display device that displays an operation screen, an optical plate that projects the operation screen into the air to generate an aerial projection surface, and a sensor device for detecting the position of an object that contacts the aerial projection surface in a three-dimensional space And a computer for specifying an operation input performed on the operation screen, and information for specifying the two-dimensional coordinates of the object in a sensing area, and from the sensor device, the sensor device In the operation input device that outputs sensor data including the depth to the object,
In the computer,
(A) detecting a position of the object on the aerial projection plane from the sensor data;
(B) when detecting from the sensor data that a part of the object is located on the sensor device side of the aerial projection plane, setting a figure surrounding the part of the object in the sensor data; When,
(C) using the depth at the set outer edge of the figure to correct the position of the object on the aerial projection plane, and recording a program including a command to be executed;
It is characterized by that.

As described above, according to the present invention, when a user performs an input operation by touching a screen displayed in the air, it is possible to suppress a decrease in detection accuracy of a touch position due to a user's erroneous operation.

FIG. 1 is a configuration diagram showing the configuration of the operation input device according to the embodiment of the present invention. FIG. 2 is a diagram illustrating the function of the sensor device used in the operation input device according to the embodiment of the present invention. FIG. 3 is a diagram illustrating an example of a sensing result of the sensor device provided in the operation input device according to the embodiment of the present invention. FIG. 4 is a flowchart showing the operation of the operation input device according to the embodiment of the present invention. FIG. 5 is a block diagram illustrating an example of a computer that implements the control device for the operation input device according to the embodiment of the present invention.

(Embodiment)
Hereinafter, an operation input device, an operation input method, and a program according to an embodiment of the present invention will be described with reference to FIGS.

[Device configuration]
First, the configuration of the operation input device in the present embodiment will be described. FIG. 1 is a configuration diagram showing the configuration of the operation input device according to the embodiment of the present invention.

The operation input device 100 in the present embodiment shown in FIG. 1 is a device that enables an input operation by touching a screen displayed in the air. As shown in FIG. 1, the operation input device 100 includes a display device 10, an optical plate 20, a sensor device 30, and a control device 40.

Display device 10 displays an operation screen for input. The optical plate 20 projects the operation screen into the air and generates an aerial projection surface 21. The sensor device 30 is a device for detecting the position of the object 50 in contact with the aerial projection plane 21 in a three-dimensional space. In the example of FIG. 1, the object 50 is a user's finger that performs an operation input.

The sensor device 30 is disposed on the back side of the aerial projection surface 21. The sensor device 30 outputs sensor data including information for specifying the two-dimensional coordinates of the object 50 in the sensing area and the depth from the sensor device 30 to the object 50.

The control device 40 includes an operation input specifying unit 41, a figure setting unit 42, and a depth correction unit 43. The operation input specifying unit 41 detects the position of the object 50 on the aerial projection plane 21 from the sensor data output by the sensor device 30. Then, the operation input specifying unit 41 specifies an operation input performed on the operation screen according to the detected position of the object 50.

Further, when the figure setting unit 42 detects that a part of the object 50 is located on the sensor device 30 side of the aerial projection surface 21 from the sensor data, the figure setting unit 42 sets a figure surrounding the part of the object in the sensor data. To do. The depth correction unit 43 corrects the position of the object 50 on the aerial projection plane 21 using the depth of the outer edge of the graphic set by the graphic setting unit 42.

Thus, in the present embodiment, for example, when the user's fingertip is located behind the aerial projection plane, the position of the finger is corrected. Therefore, according to the present embodiment, when the user performs an input operation by touching the operation screen displayed in the air, a situation in which the detection accuracy of the touch position is lowered due to an erroneous operation by the user is avoided. Is done.

Subsequently, the configuration and function of the operation input device 100 according to the present embodiment will be described in more detail with reference to FIGS. 2 and 3. FIG. 2 is a diagram illustrating the function of the sensor device used in the operation input device according to the embodiment of the present invention. FIG. 3 is a diagram illustrating an example of a sensing result of the sensor device provided in the operation input device according to the embodiment of the present invention.

First, in the present embodiment, the display device 10 is a liquid crystal display device or the like. As the optical plate 20, the image imaging plate disclosed in Patent Document 2 described above is used. The optical plate 20 has a function of collecting the light emitted from the image displayed on the screen of the display device 10 at the same distance on the opposite side as viewed from the image imaging plate to form the same image.

The sensor device 30 is a depth sensor in the present embodiment. When sensing, the depth sensor outputs image data of an image obtained by sensing and a depth (depth) added to each pixel of the image as sensor data. In the present embodiment, as the sensor device 30, a device including a camera and a distance sensor may be used.

Then, according to the image data included in the sensor data, it is possible to specify the two-dimensional coordinates of the object 50 in the sensing area. Specifically, as shown in FIG. 2, according to the image data included in the sensor data, the position of the object 50 in the vertical direction (V-axis direction) and the horizontal direction (H-axis direction) of the image. The position can be specified. Further, according to the depth included in the sensor data, the distance between the object 50 and the sensor device 30 in the Z-axis direction can be specified. The Z axis is an axis along the normal line of the sensing surface of the sensor device 30.

In the control device 40, in the present embodiment, the operation input specifying unit 41, when receiving sensor data, from the image data and each depth added to each pixel of the image, the most distal side of the object 50 (sensor device). The position in the three-dimensional space of the (30 side) part is specified. Then, the operation input specifying unit 41 converts the specified position into a position on the aerial projection plane 21.

Specifically, the operation input specifying unit 41 first extracts the most distal portion of 50 of the object. And the operation input specific | specification part 41 specifies the position in the three-dimensional space of the extracted part from the sensor data, ie, the coordinate on an X-axis, the coordinate on a Y-axis, and a depth.

Then, as shown in FIG. 2, the positions of the vertical direction (Y-axis direction) and the horizontal direction (X-axis direction) on the aerial projection plane 21 and the V-axis direction, H-axis direction, and Z-axis direction in the sensor device 30. The relationship is predetermined. Therefore, the operation input specifying unit 41 substitutes the position of the most distal portion of the object 50 in the three-dimensional space into the conversion formula obtained from this positional relationship, thereby making the object 50 perpendicular to the aerial projection plane 21. The position in the direction and the position in the horizontal direction of the aerial projection plane 21 are detected.

When the part of the object 50 is located on the sensor device 30 side of the aerial projection surface 21 (a state indicated by a broken line in FIG. 2), the figure setting unit 42 detects that from the sensor data. Specifically, the graphic setting unit 42 determines whether or not the Z coordinate (depth) of the most distal portion of the object 50 is equal to or greater than a threshold value from the sensor data. As a result of the determination, if it is not equal to or greater than the threshold (less than the threshold), the graphic setting unit 42 determines that a part of the object 50 is located on the sensor device 30 side of the aerial projection plane 21.

Further, the threshold value is set according to the position of the object 50. For example, in FIG. 2, the threshold value when the object 50 is above the aerial projection plane 21 is smaller than the threshold value when the object 50 is below the aerial projection plane 21.

Further, as shown in FIG. 4, in the present embodiment, the graphic setting unit 42 sets a rectangle that surrounds the most distal portion of the object 50 in the sensor data. The rectangle is set in this way, although the X-axis direction of the aerial projection plane 21 matches the H-axis direction of the sensor data, but the Y-axis direction of the aerial projection plane 21 matches the V-axis direction of the sensor data. If a part of the object 50 is positioned on the sensor device 30 side of the aerial projection plane 21, an error occurs in the position in the Y-axis direction.

In the present embodiment, the depth correction unit 43 uses the depth of one of the two sides on the V-axis direction side of the set rectangle, and the depth of the object 50 detected by the operation input specifying unit 41 is determined. The position of the aerial projection plane 21 in the vertical direction is corrected.

Also, which side of the two sides on the V-axis direction side is to be used is determined according to the position of the object 50. For example, when the object 50 is on the lower side in the vertical direction with respect to the intersection of the normal line passing through the center of the sensing surface of the sensor device 30 and the aerial projection surface 21, the correction is performed by the depth of the upper side in the V-axis direction. Done. On the other hand, when the object 50 is above the intersection in the vertical direction, correction is performed based on the depth of the lower side in the V-axis direction.

Furthermore, depending on the positional relationship between the aerial projection surface 21 and the sensor device 30, a side on the H-axis direction side may be used. Further, depending on the positional relationship, a figure other than a rectangle may be set. Further, only the horizontal position of the aerial projection plane 21 of the object 50 may be corrected according to these positional relationships, or the position in both the vertical direction and the horizontal direction may be corrected. There may be.

[Device operation]
Next, the operation of the operation input device 100 in the present embodiment will be described with reference to FIG. FIG. 4 is a flowchart showing the operation of the operation input device according to the embodiment of the present invention. In the following description, FIGS. 1 to 3 are referred to as appropriate. In the present embodiment, the operation input method is implemented by operating the operation input device 100. Therefore, the description of the operation input method in the present embodiment is replaced with the following description of the operation of the operation input device 100.

First, it is assumed that the sensor device 30 continuously outputs sensor data at set intervals. Whenever sensor data is output, the control device 40 receives the sensor data and executes the following processing.

As shown in FIG. 4, first, in the control device 40, the operation input specifying unit 41 determines the most distal side (sensor device 30 side) portion of the object 50 from the image data included in the sensor data and each depth. Extract (step A1).

Next, the operation input specifying unit 41 specifies the position of the part specified in step A1 in the three-dimensional space, and detects the position on the aerial projection plane 21 of the most distal part of the object 50 from the specified position. (Step A2).

Specifically, the operation input specifying unit 41 specifies the position of the extracted portion in the three-dimensional space, that is, the coordinates on the X axis, the coordinates on the Y axis, and the depth from the sensor data. Then, the operation input specifying unit 41 substitutes the specified position in the three-dimensional space for the conversion formula obtained from the positional relationship, and the position of the most distal portion of the object 50 in the vertical direction of the aerial projection plane 21 The position of the aerial projection plane 21 in the horizontal direction is detected.

Next, the graphic setting unit 42 determines whether or not the Z coordinate (depth) of the most distal portion of the object 50 is greater than or equal to a threshold value from the sensor data (step A3).

If the result of determination in step A3 is that the Z coordinate is greater than or equal to the threshold value, the graphic setting unit 42 notifies the operation input specifying unit 41 of this fact. Thereby, the operation input specification part 41 specifies a user's operation input based on the position detected by step A2 (step A4).

On the other hand, if the result of determination in step A3 is that the Z coordinate is not greater than or equal to the threshold value (smaller than the threshold value), the most distal portion of the object is located on the sensor device 30 side of the aerial projection plane 21. Therefore, as shown in FIG. 3, the graphic setting unit 42 sets a rectangle that surrounds the most distal portion of the object 50 in the sensor data (step A5).

Next, the depth correction unit 43 selects one side of the set rectangle according to the position of the object 50, and uses the depth in the selected one side to detect the most distal side of the object 50 detected in step A2. The position of the part on the aerial projection plane 21 is corrected (step A6).

When step A6 is executed, the depth correction unit 43 notifies the operation input specifying unit 41 that the position has been corrected. Thereby, the operation input specification part 41 specifies a user's operation input based on the position after correction | amendment (step A4).

The above steps A1 to A4 are repeatedly executed until the operation input device 100 is stopped.

As described above, in the present embodiment, when the object 50 such as the user's fingertip is located behind the aerial projection plane 21, the position of the object 50 on the aerial projection plane 21 is corrected. Therefore, according to the present embodiment, when the user performs an input operation by touching the operation screen on the aerial projection plane 21, even if the user performs an erroneous operation, a decrease in the detection accuracy of the touch position is suppressed. .

[program]
The program in the present embodiment may be a program that causes a computer to execute steps A1 to A6 shown in FIG. By installing and executing this program on a computer, the control device 40 of the operation input device 100 in the present embodiment can be realized. In this case, the processor of the computer functions as the operation input specifying unit 41, the figure setting unit 42, and the depth correction unit 43 to perform processing.

Further, the program in the present embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as any one of the operation input specifying unit 41, the figure setting unit 42, and the depth correction unit 43, respectively.

Here, a computer that realizes the control device 40 of the operation input device 100 by executing the program according to the present embodiment will be described with reference to FIG. FIG. 5 is a block diagram illustrating an example of a computer that implements the control device for the operation input device according to the embodiment of the present invention.

As shown in FIG. 5, the computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. With. These units are connected to each other via a bus 121 so that data communication is possible. The computer 110 may include a GPU (GraphicsGraphProcessing Unit) or an FPGA (Field-Programmable Gate Array) in addition to or instead of the CPU 111.

The CPU 111 performs various operations by developing the program (code) in the present embodiment stored in the storage device 113 in the main memory 112 and executing them in a predetermined order. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory). Further, the program in the present embodiment is provided in a state of being stored in a computer-readable recording medium 120. Note that the program in the present embodiment may be distributed on the Internet connected via the communication interface 117.

Further, specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and an input device 118 such as a keyboard and a mouse. The display controller 115 is connected to the display device 119 and controls display on the display device 119.

The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, and reads a program from the recording medium 120 and writes a processing result in the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and another computer.

Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as a flexible disk, or CD- Optical recording media such as ROM (Compact Disk Read Only Memory) are listed.

It should be noted that the control device 40 in the present embodiment can be realized not by using a computer in which a program is installed but also by using hardware corresponding to each unit. Furthermore, a part of the control device 40 may be realized by a program, and the remaining part may be realized by hardware.

Some or all of the above-described embodiments can be expressed by the following (Appendix 1) to (Appendix 15), but is not limited to the following description.

(Appendix 1)
A display device that displays an operation screen, an optical plate that projects the operation screen into the air to generate an aerial projection surface, and a sensor device for detecting the position of an object that contacts the aerial projection surface in a three-dimensional space And a control device that identifies an operation input performed on the operation screen, and the sensor device includes information for identifying a two-dimensional coordinate of the object in a sensing area, and the sensor device Output sensor data including depth to the object,
The controller is
From the sensor data, detect the position of the object in the aerial projection plane,
Further, when detecting from the sensor data that a part of the object is located on the sensor device side of the aerial projection surface, a figure surrounding the part of the object is set in the sensor data; and Using the depth at the set outer edge of the figure, the position of the object on the aerial projection plane is corrected.
An operation input device characterized by that.

(Appendix 2)
The operation input device according to attachment 1, wherein
The control device sets a rectangle surrounding a part of the object in the sensor data, and uses the depth of any one side of the rectangle to determine the position of the object in the vertical direction of the aerial projection plane, and the air Correct one or both of the horizontal positions of the projection plane,
An operation input device characterized by that.

(Appendix 3)
The operation input device according to attachment 2, wherein
The one side is set based on a positional relationship between the aerial projection plane and the sensor device.
An operation input device characterized by that.

(Appendix 4)
The operation input device according to any one of appendices 1 to 3,
In the sensor data, when the depth of the portion of the object closest to the sensor device in the sensor data is smaller than a threshold value, a part of the object is located on the sensor device side of the aerial projection plane. To determine,
An operation input device characterized by that.

(Appendix 5)
The operation input device according to any one of appendices 1 to 4,
The sensor device is a depth sensor;
An operation input device characterized by that.

(Appendix 6)
A display device that displays an operation screen, an optical plate that projects the operation screen into the air to generate an aerial projection surface, and a sensor device for detecting the position of an object that contacts the aerial projection surface in a three-dimensional space And a computer for specifying an operation input performed on the operation screen, and information for the sensor device to specify the two-dimensional coordinates of the object in a sensing area, and the sensor device from the sensor device An operation input method for outputting sensor data including a depth to an object,
(A) detecting a position of the object on the aerial projection plane from the sensor data by the computer;
(B) When the computer detects from the sensor data that a part of the object is located on the sensor device side of the aerial projection surface, a figure surrounding the part of the object is displayed in the sensor data. Set, step, and
(C) correcting the position of the object on the aerial projection plane using the depth at the outer edge of the graphic set by the computer,
An operation input method characterized by that.

(Appendix 7)
The operation input method according to attachment 6, wherein
In the step (c), in the sensor data, a rectangle surrounding a part of the object is set, and the depth of any one side of the rectangle is used to position the object in the vertical direction of the aerial projection plane; And correcting one or both of the positions in the horizontal direction of the aerial projection plane,
An operation input method characterized by that.

(Appendix 8)
The operation input method according to appendix 7,
In the step (c), the one side is set based on a positional relationship between the aerial projection plane and the sensor device.
An operation input method characterized by that.

(Appendix 9)
The operation input method according to any one of appendices 6 to 8,
In the step (b), when the depth of the portion of the object closest to the sensor device in the sensor data is smaller than a threshold, a part of the object is positioned on the sensor device side of the aerial projection plane. It is determined that
An operation input method characterized by that.

(Appendix 10)
The operation input method according to any one of appendices 6 to 9, wherein
The sensor device is a depth sensor;
An operation input method characterized by that.

(Appendix 11)
A display device that displays an operation screen, an optical plate that projects the operation screen into the air to generate an aerial projection surface, and a sensor device for detecting the position of an object that contacts the aerial projection surface in a three-dimensional space And a computer for specifying an operation input performed on the operation screen, and information for specifying the two-dimensional coordinates of the object in a sensing area, and from the sensor device, the sensor device In the operation input device that outputs sensor data including the depth to the object,
In the computer,
(A) detecting a position of the object on the aerial projection plane from the sensor data;
(B) when detecting from the sensor data that a part of the object is located on the sensor device side of the aerial projection plane, setting a figure surrounding the part of the object in the sensor data; When,
(C) correcting the position of the object on the aerial projection plane using the depth at the set outer edge of the figure;
The computer-readable recording medium characterized by recording the program containing the instruction | indication which performs this.

(Appendix 12)
A computer-readable recording medium according to appendix 11,
In the step (c), in the sensor data, a rectangle surrounding a part of the object is set, and the depth of any one side of the rectangle is used to position the object in the vertical direction of the aerial projection plane; And correcting one or both of the positions in the horizontal direction of the aerial projection plane,
A computer-readable recording medium.

(Appendix 13)
A computer-readable recording medium according to appendix 12,
In the step (c), the one side is set based on a positional relationship between the aerial projection plane and the sensor device.
A computer-readable recording medium.

(Appendix 14)
A computer-readable recording medium according to any one of appendices 11 to 13,
In the step (b), when the depth of the portion of the object closest to the sensor device in the sensor data is smaller than a threshold, a part of the object is positioned on the sensor device side of the aerial projection plane. It is determined that
A computer-readable recording medium.

(Appendix 15)
A computer-readable recording medium according to any one of appendices 11 to 14,
The sensor device is a depth sensor;
A computer-readable recording medium.

The present invention has been described above with reference to the embodiments, but the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

This application claims priority based on Japanese Patent Application No. 2018-41162 filed on Mar. 7, 2018, the entire disclosure of which is incorporated herein.

As described above, according to the present invention, when a user performs an input operation by touching a screen displayed in the air, it is possible to suppress a decrease in detection accuracy of a touch position due to a user's erroneous operation. The present invention is useful in various devices that perform input on an aerial projection plane.

DESCRIPTION OF SYMBOLS 10 Display apparatus 20 Optical plate 21 Aerial projection surface 30 Sensor apparatus 40 Control apparatus 41 Operation input specific part 42 Graphic setting part 43 Depth correction part 50 Object 110 Computer 111 CPU
112 Main Memory 113 Storage Device 114 Input Interface 115 Display Controller 116 Data Reader / Writer 117 Communication Interface 118 Input Device 119 Display Device 120 Recording Medium 121 Bus

Claims

A display device that displays an operation screen, an optical plate that projects the operation screen into the air to generate an aerial projection surface, and a sensor device for detecting the position of an object that contacts the aerial projection surface in a three-dimensional space And a control device that identifies an operation input performed on the operation screen, and the sensor device includes information for identifying a two-dimensional coordinate of the object in a sensing area, and the sensor device Output sensor data including depth to the object,
The controller is
From the sensor data, detect the position of the object in the aerial projection plane,
Further, when detecting from the sensor data that a part of the object is located on the sensor device side of the aerial projection surface, a figure surrounding the part of the object is set in the sensor data; and Using the depth at the set outer edge of the figure, the position of the object on the aerial projection plane is corrected.
An operation input device characterized by that.
The operation input device according to claim 1,
The control device sets a rectangle surrounding a part of the object in the sensor data, and uses the depth of any one side of the rectangle to determine the position of the object in the vertical direction of the aerial projection plane, and the air Correct one or both of the horizontal positions of the projection plane,
An operation input device characterized by that.
The operation input device according to claim 2,
The one side is set based on a positional relationship between the aerial projection plane and the sensor device.
An operation input device characterized by that.
The operation input device according to any one of claims 1 to 3,
In the sensor data, when the depth of the portion of the object closest to the sensor device in the sensor data is smaller than a threshold value, a part of the object is located on the sensor device side of the aerial projection plane. To determine,
An operation input device characterized by that.
The operation input device according to any one of claims 1 to 4,
The sensor device is a depth sensor;
An operation input device characterized by that.
A display device that displays an operation screen, an optical plate that projects the operation screen into the air to generate an aerial projection surface, and a sensor device for detecting the position of an object that contacts the aerial projection surface in a three-dimensional space And a computer for specifying an operation input performed on the operation screen, and information for the sensor device to specify the two-dimensional coordinates of the object in a sensing area, and the sensor device from the sensor device An operation input method for outputting sensor data including a depth to an object,
(A) detecting a position of the object on the aerial projection plane from the sensor data by the computer;
(B) When the computer detects from the sensor data that a part of the object is located on the sensor device side of the aerial projection surface, a figure surrounding the part of the object is displayed in the sensor data. Set, step, and
(C) correcting the position of the object on the aerial projection plane using the depth at the outer edge of the graphic set by the computer,
An operation input method characterized by that.
The operation input method according to claim 6,
In the step (c), in the sensor data, a rectangle surrounding a part of the object is set, and the depth of any one side of the rectangle is used to position the object in the vertical direction of the aerial projection plane; And correcting one or both of the positions in the horizontal direction of the aerial projection plane,
An operation input method characterized by that.
The operation input method according to claim 7,
In the step (c), the one side is set based on a positional relationship between the aerial projection plane and the sensor device.
An operation input method characterized by that.
The operation input method according to any one of claims 6 to 8,
In the step (b), when the depth of the portion of the object closest to the sensor device in the sensor data is smaller than a threshold, a part of the object is positioned on the sensor device side of the aerial projection plane. It is determined that
An operation input method characterized by that.
The operation input method according to any one of claims 6 to 9,
The sensor device is a depth sensor;
An operation input method characterized by that.
A display device that displays an operation screen, an optical plate that projects the operation screen into the air to generate an aerial projection surface, and a sensor device for detecting the position of an object that contacts the aerial projection surface in a three-dimensional space And a computer for specifying an operation input performed on the operation screen, and information for specifying the two-dimensional coordinates of the object in a sensing area, and from the sensor device, the sensor device In the operation input device that outputs sensor data including the depth to the object,
In the computer,
(A) detecting a position of the object on the aerial projection plane from the sensor data;
(B) when detecting from the sensor data that a part of the object is located on the sensor device side of the aerial projection plane, setting a figure surrounding the part of the object in the sensor data; When,
(C) correcting the position of the object on the aerial projection plane using the depth at the set outer edge of the figure;
Records a program that includes instructions to execute
A computer-readable recording medium.
A computer-readable recording medium according to claim 11,
In the step (c), in the sensor data, a rectangle surrounding a part of the object is set, and the depth of any one side of the rectangle is used to position the object in the vertical direction of the aerial projection plane; And correcting one or both of the positions in the horizontal direction of the aerial projection plane,
A computer-readable recording medium.
A computer-readable recording medium according to claim 12,
In the step (c), the one side is set based on a positional relationship between the aerial projection plane and the sensor device.
A computer-readable recording medium.
A computer-readable recording medium according to any one of claims 11 to 13,
In the step (b), when the depth of the portion of the object closest to the sensor device in the sensor data is smaller than a threshold, a part of the object is positioned on the sensor device side of the aerial projection plane. It is determined that
A computer-readable recording medium.
A computer-readable recording medium according to any one of claims 11 to 14,
The sensor device is a depth sensor;
A computer-readable recording medium.