JP2023047853A - Information processing device, information processing system, and program - Google Patents

Abstract

An object of the present invention is to improve the operability of an input operation on an object that has become visible through a display device, as compared with the case of performing an input operation using gestures. An HMD (30) as an information processing device includes a control unit (31), in which a display control unit (101) visually recognizes an object and an input device capable of accepting an information input operation through a display device. The contact detection unit 107 detects contact between the object and the input device based on the information regarding the positions of the object and the input device that are visible through the display device. Then, the input information receiving unit 108 receives the information input to the input device as the information input to the object. [Selection drawing] Fig. 3

Description

The present invention relates to an information processing device, an information processing system, and a program.

For example, AR (augmented reality) and VR (virtual reality) technologies, which operate objects displayed on a display device such as HMD (head-mounted display) with gestures, are capable of viewing 3D models with intuitive operations. There are merits. For example, Patent Literature 1 discloses a technology that allows a user wearing an HMD to perform input operations such as tapping and pinching while overlapping a smartphone with one or more virtual objects by gestures.

Japanese Patent No. 6346585

However, when performing an input operation on an object displayed on the HMD by gestures, many users feel that the operation is more difficult than when inputting on a general-purpose device such as a personal computer, a smartphone, or a tablet.

An object of the present invention is to improve the operability of an input operation on an object that has become visible through a display device, as compared with the case of performing an input operation using gestures.

The invention described in claim 1 comprises a processor, the processor performs control to make an object and an input device capable of receiving information input operations visible through a display device, and the display device contact between the object and the input device is detected based on information about the positions of the object and the input device that have become visible through the input device, and information input to the input device when the contact is detected is accepted as information input to the object.
In the invention recited in claim 2, the processor performs control to make the input device existing in the physical space visible through the display device. processing equipment.
In the invention recited in claim 3, the processor specifies, among the input devices existing in the physical space, one to be subjected to an input operation by a user, and performs control to make it visible through the display device. 3. The information processing apparatus according to claim 2, characterized by:
According to the fourth aspect of the present invention, the processor is configured to store any one or more of data provided from the input device, image data obtained by imaging the input device, and sensing data targeted for the input device. Based on the behavior of the input device estimated from the data, the input device to be the target of the user's input operation is specified, and control is performed to make it visible through the display device. Item 3. An information processing apparatus according to item 3.
According to the invention recited in claim 5, the processor determines the user's behavior based on the behavior of the input device estimated from the matching result of the inertial measurement data (IMU data) of the input device and the three-dimensional behavior data. 5. The information processing apparatus according to claim 4, wherein the input device to be subjected to the input operation is specified and controlled to be visible through the display device.
The invention recited in claim 6 is characterized in that the processor performs control to make the input device visible through the display device worn by the user. processing equipment.
The invention recited in claim 7 is characterized in that the processor performs control to make the input device visible via a head-mounted display as the display device worn on the user's head. The information processing apparatus according to claim 6, wherein
According to the eighth aspect of the invention, the processor controls the object and the input device capable of receiving the input operation of information of one or more of text information, voice information, and image information. 2. The information processing apparatus according to claim 1, wherein control is performed to enable visual recognition via a display device.
In the invention recited in claim 9, the processor controls to make the object and the input device capable of accepting the input operation of the one or more types of information visible through the display device, 9. The information processing apparatus according to claim 8, wherein control is performed to further make visible another object that supports the input operation to the input device.
The invention according to claim 10 is characterized in that the processor performs control to make the other object visible so as to be in the vicinity of the input device or superimposed on at least a part of the input device. The information processing apparatus according to claim 9, wherein
According to the invention recited in claim 11, the processor, as the information about the position, based on information indicating outer edges of each of the object and the input device made visible through the display device, 2. The information processing apparatus according to claim 1, wherein contact between the and the input device is detected.
According to a twelfth aspect of the invention, the processor detects contact between outer edges of a virtual area surrounding the outer edge specified from information indicating the outer edge of each of the object and the input device. 12. The information processing apparatus according to claim 11, wherein the information processing apparatus detects contact with an input device.
A thirteenth aspect of the invention is characterized in that the outer edge of the virtual area is specified based on a predetermined value as a value of the distance outward from the outer edge of each of the object and the input device. 13. The information processing apparatus according to claim 12.
According to the fourteenth aspect of the invention, when the detected contact is based on a user's intentional operation, the processor converts information input to the input device into information input to the object. 2. The information processing apparatus according to claim 1, wherein the information processing apparatus receives as
The invention according to claim 15, wherein the detected contact is an operation of moving the object to touch the input device or moving the input device to touch the object by a gesture of the user. If the input is based on the user's intentional operation, the information input to the input device is accepted as the information input to the object. 14. The information processing apparatus according to 14.
According to the sixteenth aspect of the invention, when the line of sight of the user is at or near the point of contact between the object and the input device at the timing of the contact, the processor detects the user's intent. 15. The information processing apparatus according to claim 14, wherein the information input to the input device is accepted as the information input to the object based on a physical operation.
According to the seventeenth aspect of the invention, the processor performs an input function for performing the input operation specified based on the combination of the type of the object and the function of the input device. 2. The information processing apparatus according to claim 1, wherein the information is received as information input to the object.
According to the eighteenth aspect of the present invention, the processor is configured to store at least one of data provided from the input device, image data obtained by imaging the input device, and sensing data targeted for the input device. 18. The information processing apparatus according to claim 17, wherein the function of the input device is specified based on.
In the invention recited in claim 19, the processor receives information input to the input device by the input function specified by the combination stored in the database as information input to the object. 18. The information processing apparatus according to claim 17, characterized by:
According to the invention recited in claim 20, the processor further detects a distance between the object in contact with the input device based on the information regarding the positions of the object and the input device, and 2. The information processing apparatus according to claim 1, wherein when the distance is detected, even if information is input to the input device, the information is not accepted as information input to the object.
In the invention recited in claim 21, the processor, when the detected distance is based on an intentional operation by the user, even if information is input to the input device, the information is input to the input device. 21. The information processing apparatus according to claim 20, wherein the information input to the object is not accepted.
The invention recited in claim 22 is characterized in that the processor detects the contact when registration of cooperation destination candidates has been completed between the object and the input device. An information processing apparatus according to claim 1 .
According to the twenty-third aspect of the invention, the processor is configured to link between the object and the predetermined input device or the input device in which predetermined application software for connection is installed. 23. The information processing apparatus according to claim 22, wherein said contact is detected when registration of candidates has been completed.
The invention described in claim 24 comprises: display control means for controlling an object and an input device capable of receiving an information input operation to be visible through a display device; detection means for detecting contact between the object and the input device based on information on the respective positions of the object and the input device, and detecting information input to the input device when the contact is detected; and receiving means for receiving information input to the object.
The invention described in claim 25 has a function of controlling an object and an input device capable of receiving an information input operation to be visible through a display device, and a function of detecting contact between the object and the input device based on information about the respective positions of the object and the input device; It is a program for realizing the function of accepting as information input to

According to the first aspect of the present invention, an input operation to the input device is directly linked as an input operation to the object. It can be improved more than when performing an input operation.
According to the second aspect of the present invention, an input device existing in the physical space can be made visible through a display device, and an input operation to the input device can be used as an input operation to an object.
According to the third aspect of the present invention, an input device, which is the target of a user's input operation in the real space, is made visible through a display device, and a user's input operation to the input device is performed as an input operation to an object. It becomes possible to
According to the fourth aspect of the present invention, it is possible to identify whether or not the input device is the target of the user's input operation in the real space based on the behavior of the input device.
According to the fifth aspect of the present invention, inertial measurement data (IMU data) of the input device and three-dimensional behavior data are matched to determine whether or not the input device is the target of the user's input operation in the real space. It becomes possible to estimate based on the results.
According to the sixth aspect of the present invention, the object and the input device can be visually recognized through the display device worn by the user, and the user's input operation on the input device can be used as the input operation on the object.
According to the seventh aspect of the present invention, the input device can be made visible through a head-mounted display worn on the user's head, and the user's input operation to the input device can be used as the input operation to the object. It becomes possible.
According to the eighth aspect of the present invention, the input operation of any one or more of text information, voice information, and image information to the input device made visible through the display device is performed on the object. It is possible to operate.
According to the ninth aspect of the present invention, it is possible to improve the operability of the input operation to the input device that is visible through the display device.
According to the tenth aspect of the present invention, it is possible to improve the operability of the input operation to the input device that is visible through the display device.
According to the eleventh aspect of the present invention, it is possible to detect the presence or absence of contact between the object and the input device from the information indicating the respective outer edges of the object and the input device that are visible through the display device.
According to the present invention of claim 12, by easing the contact determination that the contact is detected before the outer edges of the object and the input device that have become visible through the display device are completely in contact with each other, User operability is improved.
According to the thirteenth aspect of the present invention, it is possible to arbitrarily set the degree of easing the contact determination, so that the user's operability is further improved.
According to the fourteenth aspect of the present invention, it is possible to prevent the occurrence of unintended actions due to unintended contact by the user.
According to the fifteenth aspect of the present invention, an operation of moving an object to touch an input device or a gesture of moving an input device to touch an object is defined as an intentional operation by the user, so that the user's intention is It is possible to prevent the occurrence of unintended operations due to unintentional contact.
According to the sixteenth aspect of the present invention, since the user's intention can be estimated from the information regarding the user's line of sight at the time of contact, it is possible to prevent unintended actions due to unintended contact.
According to the seventeenth aspect of the present invention, it is possible to perform an input operation using an input function specified by a combination of the type of object and the function of the input device.
According to the eighteenth aspect of the present invention, the input function of the input device can be specified from any one of data provided from the input device, image data obtained by imaging the input device, and sensing data for the input device. It becomes possible.
According to the nineteenth aspect of the present invention, the input function can be identified from the combination of the type of object stored in the database and the function of the input device, so various combinations can be handled.
According to the twentieth aspect of the present invention, the user can release the cooperation of the input operations between the object and the input device by performing the operation of separating the object and the input device that are in contact with each other.
According to the twenty-first aspect of the present invention, it is possible to prevent the occurrence of an unintended operation due to an unintended separation of the user.
According to the twenty-second aspect of the present invention, it is possible to prevent unintended connection to an input device by making it a condition that the registration of the candidate for the cooperation destination between the object and the input device is completed in advance. can be prevented.
According to the twenty-third aspect of the present invention, the registration of the candidate for the cooperation destination is completed in advance between the object and the predetermined input device or the input device in which the predetermined application software for connection is installed. Unintended connection to an input device can be prevented.
According to the twenty-fourth aspect of the present invention, it is possible to provide an information processing system that improves the operability of an input operation on an object that has become visible through a display device, compared to the case of performing an input operation using gestures.
According to the twenty-fifth aspect of the present invention, it is possible to provide a program that improves the operability of an input operation on an object that has become visible through a display device, compared to the case of performing an input operation using gestures.

It is a figure showing composition of the whole information processing system to which this embodiment is applied. It is a figure which shows the hardware constitutions of HMD. It is a figure which shows the functional structure of the control part of HMD. 4 is a flow chart showing the flow of pairing processing. 4 is a flow chart showing the flow of HMD processing. FIG. 10 is a diagram showing an example of a case where the object is an electronic document capable of accepting text input, and the input device is a tablet terminal, among specific examples of the object and the input device that are visible to the user via the HMD; be. Among specific examples of objects and input devices that can be visually recognized by the user through the HMD, an example in which the object is a PDF (Portable Document Format) document and the input device is a pen tablet that does not have a display function. It is a figure which shows. Among specific examples of objects and input devices that can be visually recognized by the user through the HMD, the objects are objects that can accept input operations for drawing with a stylus and can be shared with other users. It is a figure which shows the example in case an input device is a tablet terminal. FIG. 10 is a diagram showing an example in which the object is a software keyboard and the input device is a tablet terminal among specific examples of the object and the input device that are visible to the user via the HMD; Among specific examples of objects and input devices that can be visually recognized by the user through the HMD, the object is a three-dimensional model that can receive input for editing by three-dimensional model design software, and the input device is a notebook. 1 is a diagram showing an example in the case of a personal computer of the type; FIG. FIG. 10 is a diagram showing a specific example of an object and an input device that can be visually recognized by the user via the HMD, in which the object is a speech balloon that can accept text input, and the input device is a tablet terminal; .

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.
(Configuration of information processing system)
FIG. 1 is a diagram showing the overall configuration of an information processing system 1 to which this embodiment is applied.
The information processing system 1 includes an HMD (head mounted display) 30 having an information processing function and an input device 50 connected to a network 90, infrared communication, visible light communication, proximity wireless communication, Bluetooth (registered trademark), RFID (registered trademark). ) and UWB (Ultra Wide Band). The network 90 is, for example, a LAN (Local Area Network), the Internet, or the like.

The HMD 30 performs control to display an object and control to allow the user U to visually recognize the input device 50 existing in the physical space. The HMD 30 also detects contact between the displayed object and the input device 50 that has become visible. Then, when the contact between the object and the input device 50 is based on the intentional operation of the user U, the HMD 30 causes the object and the input device 50 to cooperate, and then the input to the input device 50 Accept the information as entered into the object.

The HMD 30 is a head-mounted information processing device having a display for displaying image information. The HMD 30 includes a transmissive type and a non-transmissive type. The transmissive HMD 30 displays an object on the display and makes the input device 50 existing in the real space visible through the display. The non-transmissive HMD 30 displays the object and the input device 50 on the display in real time.

The user U wearing the transmissive HMD 30 can simultaneously view the object displayed on the HMD 30 and the input device 50 existing in the real space through the HMD 30 . Also, the user U wearing the non-transmissive HMD 30 can visually recognize the object displayed on the HMD 30 and the input device 50 at the same time. Note that, in the present embodiment, a transmissive HMD 30 is employed. Objects displayed on the HMD 30 include, for example, document data, whiteboards, virtual keyboards, three-dimensional models, images, moving images, audio files, and the like.

The input device 50 is an information processing device that exists in the physical space together with the user U and has a function of receiving the user's U input operation. Examples of the input device 50 include smartphones, personal computers (desktop type, notebook type), tablet terminals, so-called touch sensors, pen tablets, electronic pens, smart pens, and the like.

(HMD hardware configuration)
FIG. 2 is a diagram showing the hardware configuration of the HMD 30. As shown in FIG.
The HMD 30 has a control section 31 , a memory 32 , a storage section 33 , a communication section 34 , a display section 35 , a sensor section 36 and an imaging section 37 . These units are connected by a data bus, an address bus, a PCI (Peripheral Component Interconnect) bus, or the like.

The control unit 31 is a processor that controls its own operation through execution of various types of software such as an OS (basic software) and application software (applied software). The control unit 31 is configured by, for example, a CPU (Central Processing Unit). The memory 32 is a storage area for storing various software and data used for executing the software, and is used as a work area for calculation. The memory 32 is composed of, for example, a RAM (Random Access Memory) or the like.

The storage unit 33 is a storage area for storing input data to various software, output data from various software, and the like, and stores a database for storing various information. The storage unit 33 is configured by an HDD (Hard Disk Drive), an SSD (Solid State Drive), a semiconductor memory, or the like used for storing programs and various setting data, for example. The communication unit 34 transmits and receives data via the network 90 or by a communication method such as infrared communication. The communication unit 34 transmits and receives data between the input device 50 and the outside.

The display unit 35 presents an image to the user by means of a liquid crystal display or an organic EL display, and like a normal display, there are a non-transmissive type and a transmissive type through which the user can see forward through the display. . The former is usually called a virtual reality HMD (VR-HMD), and the latter is usually called an augmented reality HMD (AR-HMD) or a mixed reality HMD (MR-HMD).

The sensor unit 36 includes various sensors such as an optical sensor such as a LiDAR (Laser Imaging Detection and Ranging) sensor and an acceleration sensor. The imaging unit 37 is configured with a camera or the like. These units are connected by a data bus, an address bus, a PCI bus, and the like.

(Hardware configuration of input device)
The hardware configuration of the input device 50 varies depending on the type of the input device 50. For example, when the input device 50 is a smartphone, a personal computer (desktop type, notebook type), or a tablet terminal, the HMD 30 shown in FIG. In addition to the hardware configuration of , it has an operation unit. The operation unit includes, for example, a keyboard, mouse, mechanical buttons and switches, and receives input operations. The operation unit also includes a touch sensor that configures a touch panel integrally with the display unit. Further, for example, when the input device 50 is a so-called touch sensor, a pen tablet, an electronic pen, a smart pen, or the like that does not have a screen, the hardware configuration of the HMD 30 shown in FIG. It has the same configuration as the configuration.

(Functional configuration of control unit of HMD)
FIG. 3 is a diagram showing the functional configuration of the control section 31 of the HMD 30. As shown in FIG.
In the control unit 31 of the HMD 30, a display control unit 101, a data acquisition unit 102, a registration determination unit 103, a behavior estimation unit 104, a device identification unit 105, a position acquisition unit 106, a contact detection unit 107, an input An information reception unit 108 and an input control unit 109 function.

The display control unit 101 performs control to make the object and the input device 50 capable of receiving an information input operation visible through the own device. Specifically, the display control unit 101 controls the display unit 35 to display an object capable of receiving an information input operation, and visually recognizes the input device 50 existing in the physical space through the display unit 35. Control to enable.

The data acquisition unit 102 acquires sensing data from various sensors of the sensor unit 36 and data of images (still images and moving images) captured by the imaging unit 37 . Sensing data from various sensors of the sensor unit 36 includes sensing data relating to movements of the user's U hands and eyes. The data of the image captured by the imaging unit 37 includes the data of the image of the user U, the data of the image of the object present in the visible area through the display unit 35, and the data of the image of the input device 50. be

The registration determination unit 103 determines the state of registration of cooperation destination candidates between the HMD 30 and the input device 50 . The 'registration status of cooperation destination candidates' refers to a state indicating whether or not a combination of a plurality of devices are mutually registered as cooperation destination candidates and are waiting for cooperation. Such a state is also referred to as a so-called "pairing" state. Once the pairing is performed, the pairing is not required for the second and subsequent connections. The registration determination unit 103 determines whether the HMD 30 and the input device 50 are paired and are in a state of waiting for cooperation based on information held by the own device and information obtained from each of the input devices 50. .

Here, pairing between the HMD 30 and the input device 50 will be described. Pairing between the HMD 30 and the input device 50 is performed, for example, in the following procedure. That is, the pairing process may be started by the user U's voice input. In this case, for example, when the user utters "pairing", the voice may be detected and the HMD 30 may output voice guidance regarding pairing. Further, for example, the HMD 30 outputs a guidance such as "Please press the pairing start button in the application software of the device to be paired", and the connection application software pre-installed in the input device 50 is activated, and pairing is performed. may be started.

In this case, any input device 50 on which application software for connection is installed may be an open connection environment in which the pairing process is started. Further, when the user U wearing the HMD 30 on the head visually recognizes the QR code (registered trademark) displayed on the input device 50 through the HMD 30, the HMD 30 detects the QR code (registered trademark) and automatically Pairing processing may be started. Also, from the standpoint of preventing unintended pairing with the input device 50, the input device 50 that can be paired with the HMD 30 may be limited, and a closed connection environment may be created in which the HMD 30 is paired in advance.

The behavior estimation unit 104 estimates behaviors of the user U and the input device 50 based on the data acquired by the data acquisition unit 102 and the data provided from the input device 50 . Among these, the behavior of the user U refers to movements of the user's U hands and eyes. Also, the behavior of the input device 50 refers to the movement of the input device 50 in the physical space. The data acquired by the data acquisition unit 102 includes, for example, data indicating the pairing state with the input device 50, sensing data of various sensors of the sensor unit 36, and images captured by the imaging unit 37 (still images and video) data, etc.

The data provided from the input device 50 includes data indicating the pairing state with the HMD 30 , data input to the input device 50 , inertial measurement data (IMU data) of the input device 50 , tertiary data of the input device 50 . The original behavior data, data regarding the position of the input device 50, and the like are included. Specifically, for example, the behavior estimating unit 104 estimates the behavior of the input device 50 from the results of matching the inertial measurement data (IMU data) of the input device 50 and the three-dimensional behavior data.

It should be noted that how the three-dimensional data of the outer shape of the input device 50 is acquired is not particularly limited. For example, it may be acquired separately based on the type of input device pre-stored in the database of the storage unit 13 . Further, for example, the user U wearing the HMD 30 performs an action of looking at the input device 50 from all directions, and at the same time, the HMD 30 generates three-dimensional data from data of a plurality of captured images obtained by capturing images of the input device 50. You may In this case, the HMD 30 may output to the user U a voice guidance regarding the procedure for viewing the input device 50 .

The device specifying unit 105 uses the data acquired by the data acquiring unit 102, the data provided from the input device 50, the result of estimation by the behavior estimation unit 104, and the input device information stored in advance in the database of the storage unit 13. The input device 50 targeted for input operation by the user U is specified based on the combination of type and function. Specification of the input device 50 includes specification of the function of the input device 50 . The data acquired by the data acquisition unit 102 includes sensing data, image data, and the like. The data provided from the input device 50 includes data indicating the pairing state, input data, inertial measurement data (IMU data), three-dimensional behavior data, position data, and the like. Specifically, for example, the device specifying unit 105 specifies the input device 50 based on image data indicating an area visible to the user U through the own device, among the data acquired by the data acquiring unit 102. do.

The position acquisition unit 106 acquires information about the positions of each of the object and the input device 50 in the area visible to the user U through the device itself. Specifically, the position obtaining unit 106 obtains the image data obtained by the data obtaining unit 102 from the image data indicating the area that the user U can visually recognize through the display unit 35. Obtain information about the position of each of the object and the input device 50 in the area visible via 35 .

The contact detection unit 107 detects contact between an object and the input device 50 . Specifically, based on the information about the positions of the object and the input device 50 acquired by the position acquisition unit 106, the contact detection unit 107 detects the object in the area visible through the display unit 35 and the input device. A contact with the device 50 is detected. The contact between the object and the input device 50 may be when the outer edges of the object and the input device 50 are in contact with each other, or when the outer edges of the virtual area surrounding the outer edges of the input device 50 and the object are in contact with each other. good too. In this case, the “virtual area surrounding the outer edge” is specified based on a predetermined value as the value of the outward distance from the outer edge of each of the object and the input device 50 .

Further, the contact detection unit 107 detects whether or not the contact between the object and the input device 50 is based on the user U's intentional operation. The contact detection unit 107 detects contact between an object and the input device 50, and when the contact is based on an intentional operation by the user U, the object and the input device 50 are linked. . For example, when an action of grabbing an object displayed on the HMD 30 and touching the input device 50 is performed as a gesture of the user U wearing the HMD 30 , the contact detection unit 107 detects that the contact is intentional by the user U. Detect that it is based on manipulation. Further, for example, when it is detected that the line of sight of the user U at the timing of contact between the object and the input device 50 is at or near the contact point between the object and the input device 50, the contact detection unit 107 , it may be detected that the contact between the object and the input device 50 is based on the user U's intentional operation.

Further, when the contact detection unit 107 detects contact between an object and the input device 50, the contact detection unit 107 detects the state of the object. If the state of the object detected by the contact detection unit 107 is a write-protection measure that prevents input, or if the input function of the paired input device 50 does not allow input to the object, the object touches the input device 50, it is repelled without being linked. By doing so, the user U can intuitively perceive that writing is prohibited. Further, for example, the user U can notify that the input to the object is impossible by displaying on the HMD 30 a color or a pop-up message indicating that the input to the object is impossible, by outputting it by voice, or by generating vibration. It may be perceivable visually, audibly, or tactilely.

Further, the contact detection unit 107 further detects the distance between the object in contact with the input device based on the information regarding the positions of the object and the input device. As a result, the cooperation between the object and the input device 50 is released. Further, the contact detection unit 107 detects whether or not the distance between the object in contact with the input device 50 is based on the user U's intentional operation. When the contact detection unit 107 detects the distance between the object and the input device 50 and the contact is based on the intentional operation of the user U, the cooperation between the object and the input device 50 is released. becomes.

For example, when the gesture of the user U wearing the HMD 30 is an action of grabbing an object that is visible through the HMD 30 and separating it from the input device 50 , the contact detection unit 107 detects the object and the input device 50 . It is detected that the distance from is based on user U's intentional operation. Further, for example, when it is detected that the line of sight of the user U at the timing when the object and the input device 50 separate is at or near the contact point between the object and the input device 50, the contact detection unit 107 , the distance between the object and the input device 50 may be detected as being based on the user U's intentional operation.

When the contact detection unit 107 detects contact between the object and the input device 50 and the object and the input device 50 are linked, the input information reception unit 108 inputs the information input to the input device 50 to the object. accepted as information provided. That is, once the object and the input device 50 are linked, the user U can concentrate on the input operation without paying particular attention to the positional relationship between the object and the input device 50 .

Specifically, the input information receiving unit 108 receives input by an input function for performing an input operation specified by a combination of the type of object stored in the database of the storage unit 13 and the function of the input device 50. Accept the information as entered into the object. For example, when the input device 50 is a smart phone or a tablet terminal, a touch panel, a smart pen, or the like may be used as an input function for performing an input operation. Moreover, when the input device 50 is a personal computer, a keyboard, a mouse, and the like can be used.

In addition, when the contact detection unit 107 detects separation between the object and the input device 50 that are in contact with each other and the input device 50 is released, the input information reception unit 108 detects that the object and the input device 50 are disconnected from each other. , even if information is input to the input device 50, this information is not accepted as information input to the object. However, when the separation between the object and the input device 50 is detected by the contact detection unit 107, if the cause is not due to the gesture of the user U, the separation is not based on the intentional operation of the user U. It is detected that there is no object, and the cooperation between the object and the input device 50 is continued without being canceled. Therefore, the information input to the input device 50 is continuously accepted as information input to the object.

The input control unit 109 controls inputting the input information received by the input information receiving unit 108 into the object. For example, if the object is an electronic document capable of accepting text input and the input device 50 is a tablet terminal, control is performed to input the text information input to the tablet terminal into the electronic document. Further, for example, when the object is a PDF document and the input device 50 is a pen tablet, control is performed to input handwriting data, which is drawing information input to the pen tablet with a stylus, into the electronic document. Further, for example, when the object is a three-dimensional model and the input device 50 is a personal computer, control is performed to reflect input information in the three-dimensional model in order to edit the three-dimensional model.

(Pairing process)
FIG. 4 is a diagram showing the flow of pairing processing between the HMD 30 and the input device 50. As shown in FIG. FIG. 4 shows, as an example of pairing processing, a case where pairing processing is started by a user's voice input.

When the HMD 30 and the input device 50 are not paired and the voice input for instructing the start of pairing processing is performed (YES in step 201), the HMD 30 outputs voice guidance regarding the pairing. (step 202). On the other hand, if the voice input for instructing the start of the pairing process has not been performed (NO in step 201), the voice input for instructing the start of the pairing process is performed. The process of step 201 is repeated.

When an input operation for starting the pairing process is performed on the input device 50 according to the guidance from the HMD 30 (YES in step 203), the pairing start notification is sent from the input device 50 to the surrounding devices. is sent (step 204). On the other hand, if an input operation for starting the pairing process has not been performed on the input device 50 (NO in step 203), an input operation for starting the pairing process has been performed. The process of step 203 is repeated until the input device 50 is processed.

When the pairing start notification transmitted from the input device 50 is received by the HMD 30 within a predetermined time (YES in step 205), the HMD 30 sends an input for confirming the pairing to the user U. Guidance on the operation is notified (step 206). For example, guidance such as "Please enter your passcode and press the pairing confirmation button" is notified. On the other hand, if the transmitted pairing start notification is not received by the HMD 30 within the predetermined time (NO in step 205), the process ends without pairing.

Here, when the user U performs an operation according to the notified content (YES in step 207), the input device 50 transmits a pairing confirmation notification including a passcode to the surrounding devices (step 208). . On the other hand, if the user U does not perform the notified content operation (NO in step 207), the process of step 207 is repeated until the user U performs the notified content operation.

When the transmitted pairing confirmation notification is received by HMD 30 within a predetermined period of time (YES in step 209), HMD 30 transmits data that can identify input device 50 and input A request is made to send data regarding the features that device 50 can provide (step 210). Data that can identify the input device 50 includes, for example, the Wi-Fi address of the input device 50 . On the other hand, if the transmitted pairing confirmation notification is not received by the HMD 30 within the predetermined time (NO in step 209), the process ends without pairing.

When the input device 50 transmits the data requested by the HMD 30 to the HMD 30 (YES in step 211), the HMD 30 receives and stores the data transmitted from the input device 50 (step 212). On the other hand, if the input device 50 has not sent the data requested by the HMD 30 to the HMD 30 (NO in step 211), the input device 50 sends the data requested by the HMD 30 to the HMD 30. The process of step 211 is repeated until the data is transmitted.

Then, the HMD 30 notifies the user U of a guide for acquiring the three-dimensional data of the outer shape of the input device 50 (step 213). For example, a voice guidance such as "Please hold the device (input device 50) in your hand and look at the device from various angles" is notified. Therefore, when the user U grasps the input device 50 and gazes at it while changing its orientation and holding it in different directions (YES at step 214 ), the HMD 30 displays the characteristics of the external shape of the input device 50 . The recognition results are stored as features of the external shape of the paired input device 50 (step 215).

Specifically, the HMD 30 estimates the three-dimensional shape of the input device 50 held by the user U from the vicinity of the point the user U gazes at, and recognizes the features of the external shape of the input device 50 from various angles. Then, the recognition result is stored as the feature of the external shape of the paired input device 50 . As a result, the pairing between the HMD 30 and the input device 50 is completed, and a state of waiting for cooperation is established. On the other hand, when the user U holds the input device 50 and does not look at it from all directions by changing the orientation or changing the grip while gazing at it (NO in step 214), this The process of step 214 is repeated until such an operation is performed.

(Processing of HMD)
FIG. 5 is a flow chart showing the flow of cooperation processing by the HMD 30. As shown in FIG.
The HMD 30 performs control to make the object and the physical space visible through itself (step 401). Specifically, control is performed to display the object on the display unit 35 of the HMD 30 . In this embodiment, since the transmissive HMD 30 is employed, the input device 50 is visible through the display section 35 .

The HMD 30 recognizes an object present in the user's field of view from the data of the image captured by the imaging section 37 (step 402). As a result of specifying the input device 50, if the paired input device 50 exists in the field of view of the user U (YES in step 403), the state is awaiting cooperation (step 404). The HMD 30 estimates the behavior of each of the user U and the input device 50 based on the data acquired by the data acquisition unit 102 and the data acquired from each of the input devices 50 (step 405). On the other hand, if there is no paired input device within the field of view of the user (NO in step 403), the process returns to step 402. FIG.

The HMD 30 obtains information about the positions of the object and the input device 50 from the data of the image showing the area that the user U can visually recognize through the HMD 30 (step 406). When the HMD 30 detects contact between the object and the input device 50 in an area visible through the HMD 30 based on the obtained information regarding the respective positions of the object and the input device 50 (YES in step 407), the storage unit An input method and a display method are determined (step 408) based on the combination of the type and function of the input device 50 pre-stored in the 13 databases. Then, the HMD 30 receives the information input to the input device 50 as information input to the object (step 409), and performs control to input the received input information to the object (step 410). On the other hand, if contact between the object and the input device 50 has not been detected (NO in step 407), the HMD 30 repeats the process of step 407 until contact between the object and the input device 50 is detected.

(Concrete example)
6 to 11 are diagrams showing specific examples of objects and the input device 50 that the user U can visually recognize via the HMD 30. FIG. It is assumed that the HMD 30 and the input device 50 are paired in advance.
FIGS. 6A to 6C show an example in which the object V1 made visible through the HMD 30 is an electronic document capable of accepting text input, and the input device 50 is a tablet terminal. It is FIG. 6A shows a state in which the object V1 that has become visible through the HMD 30 and the input device 50 are separated and are not linked. Here, when the user U performs an action of grabbing the object V1 and moving the object V1 as it is to touch the input device 50, contact between the object V1 and the input device 50 is detected.

When contact between the object V1 and the input device 50 is detected as a result of the user U's gesture, the state in which the object V1 and the input device 50 are in contact, i.e., the state in which the input device 50 is in contact, as shown in FIG. state. Then, a software keyboard is displayed so as to be superimposed on the input device 50 that is visible through the HMD 30 . This enables the user U to perform an input operation on the displayed software keyboard.

Then, when the user U performs an input operation on the input device 50 existing in the real space while looking at the software keyboard displayed on the HMD 30, the input operation is linked to the input operation of the software keyboard. As a result, information input to the input device 50 is accepted as information input to the object V1, and reflected in the object V1 in a visible manner. Note that in FIG. 6B , an input operation is performed on a software keyboard displayed so as to be superimposed on the input device 50 that is visible through the HMD 30 . The configuration may be such that an input operation is performed with respect to a software keyboard displayed on the display unit.

When the user U completes the input operation on the input device 50, the user U gestures to press the end button B1 shown in FIG. 6B. Then, as shown in FIG. 6C, the object V1 and the input device 50, which are in contact and linked together, are separated from each other. As a result, the object V1 and the input device 50 are not linked, and the software keyboard superimposed on the input device 50 disappears. The input information is reflected in the object V1 and displayed in a visible manner.

It should be noted that the method of separating the object V1 and the input device 50 that are in contact and in cooperation with each other to put them in a non-cooperation state is not limited to the method of pressing the end button B1. For example, if the user U makes a gesture to touch the input device 50 to grab the object V1 that is in a linked state, and then move the object V1 as it is to move away from the input device 50, the unlinked state can also be achieved. can do.

FIGS. 7A to 7C show an example in which the object V2 made visible through the HMD 30 is a PDF document, and the input device 50 is a pen tablet that does not have a display function. ing. FIG. 7A shows a state in which the object V2 that has become visible through the HMD 30 and the input device 50 are separated and are not linked. Here, when the user U performs an action of grabbing the object V2 and moving the object V2 as it is to touch the input device 50, contact between the object V2 and the input device 50 is detected.

When contact between the object V2 and the input device 50 is detected as a result of the user U's gesture, the state in which the object V2 and the input device 50 are in contact, that is, the state in which the input device 50 is in contact, as shown in FIG. state. Then, the part to be edited in the PDF document is enlarged and displayed so as to be superimposed on the input device 50 that is visible through the HMD 30 . As a result, the user U can draw handwriting with the stylus as an input operation on the enlarged display.

Then, when the user U performs handwriting drawing with a stylus as an input operation on the input device 50 existing in the real space while looking at the portion enlarged and displayed so as to be superimposed on the input device 50, the input operation is as follows. It is interlocked with the input operation of the object V2. As a result, information input to the input device 50 is accepted as information input to the object V2, and is reflected in the object V2 in a visible manner.

When the user U completes the input operation on the input device 50, the user U gestures to press the end button B2 shown in FIG. 7B with the stylus. Then, as shown in FIG. 7C, the object V2 and the input device 50, which are in contact and linked together, are separated from each other. As a result, the object V2 and the input device 50 are not linked, and the portion enlarged and displayed so as to be superimposed on the input device 50 disappears. Further, the input information is displayed on the object V2 in a visually recognizable manner.

It should be noted that the method of separating the object V2 and the input device 50 that are in contact and in cooperation with each other to bring them into a non-cooperation state is not limited to the method of pressing the end button B2. For example, by a gesture of the user U, the object V2 that is in a state of contact with the input device 50 and is in cooperation is grabbed, and the object V2 is moved as it is to move away from the input device 50. can do.

In FIGS. 8A to 8C, an object V3 that has become visible through the HMD 30 can accept an input operation for drawing with a stylus and can be shared with other users U. An example is shown in which the input device 50 is an object (hereinafter referred to as a “whiteboard”) and the input device 50 is a tablet terminal. FIG. 8A shows a state in which the object V3 that has become visible through the HMD 30 and the input device 50 are separated and are not linked. Here, as a gesture of the user U, if the object V3 is indicated by the pointer L displayed by the pinching operation, and the object V3 is moved and brought into contact with the input device 50, the object V3 and the input device 50 are moved. A touch is detected.

When contact between the object V3 and the input device 50 is detected as a result of the user U's gesture, the object V3 returns to its original position and the object V3 and the input device 50 are separated as shown in FIG. 8B. Be in a collaborative state. Then, an image (whiteboard image) of the object V3 is displayed so as to be superimposed on the input device 50 that is visible through the HMD 30 . This enables the user U to draw by hand with the stylus as an input operation for the displayed image of the object V3.

Then, when the user U performs handwriting drawing with a stylus as an input operation on the input device 50 existing in the real space while viewing the image of the object V3 displayed so as to be superimposed on the input device 50, the input operation is performed. is interlocked with the input operation of the object V3. As a result, information input to the input device 50 is accepted as information input to the object V3, and reflected in the object V3 in a visible manner.

When the user U completes the input operation on the input device 50, the user U gestures to press the end button B3 shown in FIG. 8B with the stylus. Then, the object V3 and the input device 50 are not linked, and the image (whiteboard image) of the object V3 superimposed on the input device 50 disappears as shown in FIG. 8(C). . Further, the input information is displayed on the object V3 in a visually recognizable manner.

It should be noted that the method of separating the object V3 and the input device 50 that are in contact and in cooperation with each other to put them in a non-cooperation state is not limited to the method of pressing the end button B3. For example, the gesture of the user U may indicate the object V3 that is in contact with the input device 50 and in cooperation with the pointer L, and move the object V3 as it is to separate it from the input device 50. can be left without.

FIGS. 9A to 9C show an example in which the object V4 made visible through the HMD 30 is a software keyboard and the input device 50 is a tablet terminal. FIG. 9A shows a state in which the object V4 that has become visible through the HMD 30 and the input device 50 are separated and are not linked. In FIG. 9A, the object V5 reflecting the information input to the object V4 is visible through the HMD 30. In FIG. Here, when the user U performs an action of grabbing the object V4 and moving the object V4 as it is to touch the input device 50, contact between the object V4 and the input device 50 is detected.

When contact between the object V4 and the input device 50 is detected as a result of the user U's gesture, the object V4 as the software keyboard disappears and becomes visible through the HMD 30, as shown in FIG. 9B. A software keyboard is displayed so as to be superimposed on the input device 50 . Also, an object V6 appears as another object so as to touch the input device 50. FIG. Input lines and conversion candidates are displayed in object V6. Therefore, the user U can perform an input operation on the software keyboard displayed so as to be superimposed on the input device 50 while looking at the input lines and conversion candidates displayed on the object V6.

When the user U performs an input operation on the software keyboard while viewing the input lines and conversion candidates displayed in the object V6, the input information is accepted as the information input to the object V5. Then, it is reflected in the object V5 in a visible manner. Note that FIG. 9B shows a configuration in which an input operation is performed on the software keyboard displayed so as to be superimposed on the input device 50, but an input operation on the software keyboard actually displayed on the input device 50 is performed. It may be a configuration that is performed.

When the user U completes the input operation on the input device 50, the user U gestures to press the end button B4 shown in FIG. 9B. Then, as shown in FIG. 9C, the object V6 disappears, the object V6 and the input device 50 are not linked, and the software keyboard superimposed on the input device 50 disappears. In addition, the information input by the software keyboard is displayed on the object V5 in a visually recognizable manner.

It should be noted that the method of separating the object V6 and the input device 50 which are in contact and in cooperation with each other and putting them into a non-cooperation state is not limited to the method of pressing the end button B4. For example, by a gesture of the user U, the object V4 that is in a linked state by contacting the input device 50 is grabbed, and the object V4 is moved as it is to move away from the input device 50. can do.

In FIGS. 10A to 10C, an object V7 made visible through the HMD 30 is a three-dimensional model capable of accepting input for editing by the three-dimensional model design software, and the input device 50 is a notebook type personal computer. FIG. 10A shows a state in which the object V7 that has become visible through the HMD 30 and the input device 50 are separated and are not linked. Here, when the user U performs an action of grabbing the object V7 and moving the object V7 as it is to touch the input device 50, contact between the object V7 and the input device 50 is detected.

When contact between the object V7 and the input device 50 is detected as a result of the user U's gesture, the object V7 returns to its original position and is visible through the HMD 30, as shown in FIG. 10B. The editing screen of the three-dimensional model is displayed on the display section of the input device 50 that is set to . As a result, the user U can perform an input operation for editing the object V7, which is a three-dimensional model, while viewing the edit screen of the three-dimensional model that has become visible through the HMD 30. FIG. When the user U performs an input operation for editing the 3D model while viewing the 3D model editing screen, the 3D model reflecting the input information is displayed as an object V7.

When the user U completes the input operation on the input device 50, the user U gestures to press the end button B5 shown in FIG. 10B. Then, as shown in FIG. 10C, the object V7 and the input device 50 are not linked, and the editing screen of the three-dimensional model displayed on the display section of the input device 50 disappears. Also, the three-dimensional model reflecting the information input for editing the three-dimensional model is displayed in a visually recognizable form as an object V7.

FIGS. 11A to 11C show an example in which the object V8 made visible through the HMD 30 is a balloon capable of accepting text input, and the input device 50 is a tablet terminal. ing. The balloons in FIGS. 11A to 11C are balloons displayed so as to come into contact with the inspection equipment in the real space, which is made visible through the HMD 30, or in the vicinity of the inspection equipment. Information indicating the inspection facility itself, information indicating the details of the inspection, information obtained from the inspection facility, and the like are displayed in an editable manner in this balloon. FIG. 11A shows a state in which the object V8 that has become visible through the HMD 30 and the input device 50 are separated and are not linked. Here, when the user U performs an action of grabbing the object V8 and moving the object V8 to contact the input device 50, contact between the object V8 and the input device 50 is detected.

When contact between the object V8 and the input device 50 is detected as a result of the user U's gesture, the object V8 returns to its original position and is visible through the HMD 30, as shown in FIG. 11B. A software keyboard is displayed so as to be superimposed on the input device 50 that is set to . Also, an object V9 appears as another object so as to touch the input device 50. FIG. An input line and conversion candidates are displayed in object V9. As a result, the user U can perform an input operation on the software keyboard superimposed on the input device 50 while viewing the input line and conversion candidates displayed on the object V9.

When the user U performs an input operation on the software keyboard while looking at the input lines and conversion candidates displayed in the object V9, the input information is accepted as the information input to the object V9 and is visible. is reflected in the object V8. Note that FIG. 11B shows a configuration in which an input operation is performed on the software keyboard displayed so as to be superimposed on the input device 50, but an input operation on the software keyboard actually displayed on the input device 50 is performed. It may be a configuration that is performed.

When the user U completes the input operation on the input device 50, the user U gestures to press the end button B6 shown in FIG. 11B. Then, as shown in FIG. 11C, the object V9 disappears, the object V9 and the input device 50 are not linked, and the software keyboard superimposed on the input device 50 disappears. In addition, the information input from the software keyboard is displayed in the object V8 in a visually recognizable manner.

It should be noted that the method of putting the object V8 in the linked state and the input device 50 into the non-linked state is not limited to the method of pressing the end button B6. For example, the uncoordinated state can also be achieved by the user U gesturing to grab the object V9 that is in contact with the input device 50, and moving the object V9 as it is to separate it from the input device 50. can.

Although the embodiment has been described above, the present invention is not limited to the embodiment described above. Moreover, the effects of the present invention are not limited to those described in the above embodiment. For example, the system configuration shown in FIG. 1 and the hardware configuration shown in FIG. 2 are merely examples for achieving the object of the present invention, and are not particularly limited. Also, the functional configuration shown in FIG. 3 is merely an example, and is not particularly limited. It is sufficient that the information processing system 1 of FIG. 1 has a function capable of executing the above-described processing as a whole, and the functional configuration used to realize this function is not limited to the example of FIG.

Also, the order of the processing steps shown in FIGS. 4 and 5 is merely an example and is not particularly limited. In addition to the processing performed chronologically along the order of the illustrated steps, the steps may not necessarily be processed chronologically, but may be performed in parallel or individually. Further, the specific examples shown in FIGS. 6 to 11 are only examples and are not particularly limited.

Further, for example, in FIGS. 6 to 11 , the gesture of the user U is configured to touch the input device 50 by grasping or showing an object, but the present invention is not limited to this. As long as the object and the input device 50 come into contact with each other, the user's U gesture may be such that the input device 50 in the physical space is moved to come into contact with the object. Specifically, the user U moves the hand-held input device 50 to touch the object displayed on the HMD 30 . In this case, the contact detection unit 107 (see FIG. 3) of the HMD 30 detects that the contact between the object and the input device 50 is based on the user's U intentional operation (gesture in this case).

Further, for example, although the transmissive HMD 30 is employed in the above-described embodiment, a non-transmissive HMD 30 may be employed. That is, in the present invention, it is sufficient that the user can visually recognize the object and the input device through the display device. For example, the captured image of the input device may be displayed on the non-transmissive HMD in real time.

DESCRIPTION OF SYMBOLS 1... Information processing system 30... HMD 31... Control part 50... Input device 90... Network 101... Display control part 102... Data acquisition part 103... Registration determination part 104... Behavior estimation part 105... Device identification unit 106 Position acquisition unit 107 Contact detection unit 108 Input information reception unit 109 Input control unit

Claims (25)

with a processor
The processor
Control to make the object and input device capable of accepting information input operations visible through the display device,
Detecting contact between the object and the input device based on information about the respective positions of the object and the input device that have become visible through the display device;
When the contact is detected, the information input to the input device is accepted as the information input to the object,
Information processing equipment. The processor controls the input device existing in the physical space to be visible through the display device,
The information processing device according to claim 1 . The processor specifies, among the input devices existing in the physical space, the target of the user's input operation, and performs control to make it visible through the display device,
The information processing apparatus according to claim 2. The processor estimates the behavior of the input device from at least one of data provided from the input device, image data obtained by imaging the input device, and sensing data targeted for the input device. Based on, the input device that is the target of the user's input operation is specified and controlled to be visible through the display device.
The information processing apparatus according to claim 3. The processor controls the input device to be operated by the user based on the behavior of the input device estimated from the result of matching the inertial measurement data (IMU data) of the input device and the three-dimensional behavior data. is specified and controlled to be visible through the display device,
The information processing apparatus according to claim 4. The processor performs control to make the input device visible via the display device worn by the user,
The information processing device according to claim 1 . The processor performs control to make the input device visible via a head-mounted display as the display device mounted on the user's head,
The information processing device according to claim 6 . The processor controls to make the object and the input device capable of accepting the input operation of information of one or more types of text information, voice information, and image information visible through the display device. characterized by performing
The information processing device according to claim 1 . The processor supports the input operation to the input device as control to make the object and the input device capable of accepting the input operation of the one or more types of information visible through the display device. characterized by performing control to make other objects to be more visible,
The information processing apparatus according to claim 8 . The processor performs control to make the other object visible so as to be near the input device or overlap at least a part of the input device,
The information processing apparatus according to claim 9 . The processor detects contact between the object and the input device based on information indicating outer edges of the object and the input device that are visible through the display device as the position information. characterized by
The information processing device according to claim 1 . The processor detects, as contact between the object and the input device, contact between outer edges of a virtual area surrounding the outer edges, which is specified from information indicating the outer edges of each of the object and the input device. characterized by
The information processing device according to claim 11 . The outer edge of the virtual area is specified based on a predetermined value as a value of the distance outward from the outer edge of each of the object and the input device,
The information processing apparatus according to claim 12. When the detected contact is based on a user's intentional operation, the processor accepts information input to the input device as information input to the object,
The information processing device according to claim 1 . When the detected contact is an operation of moving the object to contact the input device or an operation of moving the input device to contact the object, the detected contact is a gesture of the user. is based on the user's intentional operation, and accepts information input to the input device as information input to the object,
The information processing apparatus according to claim 14. When the line of sight of the user is at or near a contact point between the object and the input device at the timing of the contact, the processor determines that the input is based on an intentional operation by the user. characterized by accepting information input to the device as information input to the object,
The information processing apparatus according to claim 14. The processor converts information input to the input device by an input function for performing the input operation specified based on the combination of the type of the object and the function of the input device to information input to the object. characterized by accepting as
The information processing device according to claim 1 . The processor controls the function of the input device based on any one or more of data provided from the input device, image data obtained by imaging the input device, and sensing data targeted for the input device. characterized by identifying
The information processing apparatus according to claim 17. The processor receives information input to the input device by the input function specified by the combination stored in the database as information input to the object,
The information processing apparatus according to claim 17. The processor
further detecting a distance between the object in contact with the input device based on the information about the positions of each of the object and the input device;
When the separation is detected, even if information is input to the input device, the information is not accepted as information input to the object,
The information processing device according to claim 1 . When the detected distance is based on a user's intentional operation, the processor does not accept information input to the input device as information input to the object. characterized by
The information processing apparatus according to claim 20. The processor detects the contact when the registration of the candidate for the cooperation destination is completed between the object and the input device,
The information processing device according to claim 1 . The processor, when registration of candidates for cooperation destinations has been completed between the object and the predetermined input device or the input device in which predetermined application software for connection is installed, , characterized by detecting the contact,
The information processing apparatus according to claim 22. display control means for performing control to make an object and an input device capable of accepting an information input operation visible through a display device;
detection means for detecting contact between the object and the input device based on information on the respective positions of the object and the input device that are visible through the display device;
receiving means for receiving information input to the input device as information input to the object when the contact is detected;
An information processing system characterized by having: A function of performing control to make an object and an input device capable of accepting an information input operation visible through a display device;
a function of detecting contact between the object and the input device based on information about the respective positions of the object and the input device that are visible through the display device;
a function of receiving information input to the input device as information input to the object when the contact is detected;
program to make it happen.

Images