JP7094280B2 - Methods, systems and non-transient computer-readable recording media to assist in controlling objects - Google Patents

Description

The present invention relates to methods, systems and non-transient computer readable recording media for assisting control of an object.

Recently, techniques have been introduced to help users control real objects such as home appliances and virtual objects such as avatars based on gestures.

In particular, recently, as interest in augmented reality (AR) and virtual reality (VR) has increased and research and development in related technical fields have been actively carried out, users have become real or virtual reality. The reality is that there is an increasing demand for technologies that support the smooth control of interactable objects that exist in.

Conventionally, there is a technique that allows a user's control command to be transmitted only to a target object specified in advance by the user or system, and a user can indicate the target object with a pointer such as a mouse cursor or the target object. Used for the target object by performing explicit actions such as speaking the name of the object separately (for example, "turn off the lights in the living room", "please lower the temperature of the air conditioner of No. 301", etc.) There is a place where the technique to make the control command of the person transmitted is introduced.

However, according to the above-mentioned prior art, the user cannot control an object other than the predetermined object, or each time the object is controlled, the object to be controlled is directly specified. Due to the inconvenience that must be done, there is a limit that it is inflexible and inconvenient in a practical usage environment.

In particular, it is common for various types of objects (for example, home appliance devices, furniture, etc.) to exist in indoor spaces such as offices and houses, but these objects have attributes such as size and function. Since they are different, the method of controlling the object also needs to be adaptively set according to the attributes of the object. For example, turning the light on and off can be controlled by the gesture of pressing and turning the power switch of the light, and opening and closing the window can be controlled by the gesture of sliding the window up and down or left and right. However, according to the technology introduced in the past, it is difficult to effectively control various types of objects because the object can only be controlled through a uniform control method that does not reflect the attributes of the object. There are limits.

Therefore, the present inventor uses a control vector specified in a real-world coordinate system according to a gesture or operation intuitively performed by the user using his / her body part or a predetermined vector control means. We propose a technology that enables the user to accurately control various objects by performing intuitive gestures or operations by mapping them with the control coordinates on the object reference coordinate system defined by the attributes. I'm about to do it.

An object of the present invention is to solve all the above-mentioned problems.

Further, the present invention specifies a control vector (vector) corresponding to a gesture or operation performed by the user in a real-world coordinate system, and the object indicated by the control vector is controlled by the gesture or operation performed by the user. It is determined as the target object and is specified by mapping by performing mapping to specify the control coordinates corresponding to the size or direction of the control vector in the target object reference coordinate system defined based on the attributes of the target object. By referring to the control coordinates to determine the control commands that apply to the target object, the user simply makes intuitive gestures or operations to generate (or identify) the control vector for a variety of objects. Other objectives are to provide methods, systems and non-transient computer-readable recording media that enable accurate and convenient control.

A typical configuration of the present invention for achieving the above object is as follows.

According to one aspect of the present invention, a method for assisting the control of an object, wherein a control vector corresponding to a gesture or operation performed by a user is specified in a real-world coordinate system, the control vector. The size or direction of the control vector in the target object reference coordinate system defined based on the attributes of the target object, the step of determining the object instructed by the object as the target object controlled by the gesture or operation performed by the user. A method comprising performing a mapping that specifies the control coordinates corresponding to the object, and determining a control command applied to the object with reference to the control coordinates identified by the mapping. Provided.

According to another aspect of the present invention, a control vector (vector) corresponding to a gesture or operation performed by a user, which is a system for supporting the control of an object, is specified in a real-world coordinate system, and the control vector is described. The object indicated by is determined as the target object controlled by the gesture or operation performed by the user, and in the size or direction of the control vector in the target object reference coordinate system defined based on the attributes of the target object. An object control management unit that performs mapping to specify the corresponding control coordinates and determines a control command applied to the target object with reference to the control coordinates specified by the mapping, and the user. And a system including a spatial information management unit for storing spatial information about the object is provided.

In addition, other methods for embodying the present invention, systems and non-transient computer-readable recording media for recording computer programs for carrying out said methods are further provided.

According to the present invention, a control vector specified in a real-world coordinate system by a gesture or operation intuitively performed by a user using his / her body part or a predetermined vector control means is defined by the attribute of the corresponding object. By mapping with the control coordinates on the object reference coordinate system, the user can perform intuitive gestures or operations to generate (or specify) the control vector, and various objects can be accurately and conveniently used. The effect of being able to control is achieved.

Further, according to the present invention, when the user sees from his / her own viewpoint, the object that appears to overlap with his / her fingertip can be determined as the target object, so that there is no limitation on the effective area (that is, the controllable area) and the user can use it. The effect of gaining control over all objects in the visible area is achieved.

Further, according to the present invention, since the object indicated by the vector control means that the user can operate can be determined as the target object, there is no limitation on the effective region (that is, the controllable region), and the user can control the vector. The effect of being able to control all objects in the area that can be intuitively predicted (or estimated) as directed by is achieved.

Further, according to the present invention, since the process of determining the target object to be controlled and the process of determining the control command applied to the target object are both executed, the object to be controlled is changed (that is,). , Conversion from the first object to the second object), it is not necessary to separately change only one of the target object and the control command, and as a result, the effect of improving convenience is achieved. To.

Further, according to the present invention, since the process of determining the target object to be controlled and the process of determining the control command applied to the target object are both performed, the user does not intend to do so. However, the effect of significantly reducing the possibility that the corresponding object may perform a predetermined operation (that is, a malfunction) is achieved.

FIG. 6 is a drawing illustrating an exemplary configuration of an overall system for assisting control of an object according to an embodiment of the present invention. The drawing which shows exemplary the internal structure of the control support system of an object by one Embodiment of this invention. The drawing which shows the situation which the existence of various objects which can be controlled in the object space by one Embodiment of this invention exemplify. The drawing which shows the structure which maps the control vector of the coordinate system of a real world, and the control coordinate of a target object reference coordinate system by one Embodiment of this invention. The drawing which shows the structure which maps the control vector of the coordinate system of a real world, and the control coordinate of a target object reference coordinate system by one Embodiment of this invention. The drawing which shows the structure which maps the control vector of the coordinate system of a real world, and the control coordinate of a target object reference coordinate system by one Embodiment of this invention. The drawing which shows the structure which maps the control vector of the coordinate system of a real world, and the control coordinate of a target object reference coordinate system by one Embodiment of this invention. The drawing which shows exemplary the structure which the control vector is specified by the operation performed by the user using the vector control means by one Embodiment of this invention. The drawing which shows exemplary the structure which the control vector is specified by the operation performed by the user using the vector control means by one Embodiment of this invention.

For a detailed description of the invention described below, reference to the accompanying drawings illustrated by way of example certain embodiments in which the invention may be practiced. These examples will be described in detail so that those skilled in the art can practice the present invention. It should be understood that the various embodiments of the invention are different from each other but need not be mutually exclusive. For example, the particular shapes, structures and properties described herein may be embodied in other embodiments in relation to one embodiment without departing from the spirit and scope of the invention. It should also be understood that the location or placement of the individual components within each disclosed embodiment may be modified without departing from the spirit and scope of the invention. Therefore, the detailed description described below is not limited in meaning, and the scope of the present invention, if appropriate, is attached with all the scope equivalent to that claimed by the claim. Limited only by the term. Similar reference numerals in the drawings indicate the same or similar functions across various aspects.

Hereinafter, in order to enable a person having ordinary knowledge in the technical field to which the present invention belongs to easily carry out the present invention, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Overall system configuration

FIG. 1 is a drawing illustrating an exemplary configuration of an overall system for assisting control of an object according to an embodiment of the present invention.

As illustrated in FIG. 1, the entire system according to an embodiment of the present invention may be configured to include a communication network 100, an object control support system 200, and an object 300.

First, the communication network 100 according to an embodiment of the present invention can be configured regardless of the form of communication such as wired communication and wireless communication, and can be configured as a short-range communication network (LAN; Local Area Network), an urban area communication network ( It may be composed of various communication networks such as MAN (Metropolitan Area Network) and wide area network (WAN; Wide Area Network). Preferably, the communication network 100 referred to herein can be the known Internet or the World Wide Web (WWW). However, the communication network 100 is not limited to this, and may include a known radio data communication network, a known telephone network, or a known radio television communication network in at least a part thereof.

For example, the communication network 100 is a wireless data communication network, and is a radio frequency (RF; Radio Frequency) communication, Wi-Fi (WiFi) communication, cellular (LTE, etc.) communication, Bluetooth (registered trademark) (communication, more specifically). Can embody conventional communication methods such as low power Bluetooth® (BLE; Bluetooth® Low Energy)), infrared communication, ultrasonic communication and the like, at least in part thereof. As another example, the communication network 100 is an optical communication network, and may embody a conventional communication method such as LiFi (Lift Fidelity) communication at least in a part thereof.

Next, the object control support system 200 according to the embodiment of the present invention may be a digital device including memory means, equipped with a microprocessor, and having computing power. Such an object control support system 200 can be a server system.

To this end, the object control support system 200 identifies a control vector corresponding to a gesture or operation performed by the user in a real-world coordinate system and is instructed by the control vector, as described in detail below. The object to be controlled is determined as the target object controlled by the gesture or operation performed by the user, and the control coordinates corresponding to the size or direction of the control vector in the target object reference coordinate system defined based on the attributes of the target object are set. For the user to generate (or identify) a control vector by performing a mapping to identify and referencing the control coordinates identified by the mapping to determine the control instructions applied to the target object. By simply performing intuitive gestures or operations, it is possible to perform functions that enable accurate and convenient control of various objects. The functions of the object control support system 200 will be described in more detail below.

On the other hand, the object control support system 200 has been described as described above, but such an explanation is exemplary and requires at least a part of the functions and components required for the object control support system 200. Those skilled in the art may be realized within the object 300 (ie, device) or external system (not shown) to be controlled accordingly, or may be included within the object 300 or external system (not shown). It is self-evident.

Then, according to one embodiment of the present invention, the object 300 can perform a function as an object that can be controlled by a gesture or operation of the user. That is, according to one embodiment of the present invention, the object 300 receives a control command by a user's gesture or operation from the object control support system 200, another object 300, or an external system (not shown), and the reception is received. Control commands can be used to control one's own actions or functions. According to one embodiment of the invention, the object 300 can be a real object that exists in the real world and can exist in virtual reality or be augmented, mixed or merged with respect to the real world. It can be a virtual object.

Specifically, the object 300 according to an embodiment of the present invention may be a device including a function capable of communicating after being connected to an object control support system 200, another object 300, or an external system (not shown). Any number of devices provided with memory means, equipped with a microprocessor and capable of computing power can be adopted as the object 300 according to the present invention. Specifically, according to one embodiment of the present invention, the object 300 is a wearable device such as a smart glass, a smart watch, a smart band, a smart ring, a necklace, a smartphone, a smart pad, a desktop computer, a laptop computer, or a workstation. , PDA, web pad, mobile phone, etc. can be a little traditional device. Further, according to one embodiment of the present invention, the object 300 can be an IoT (Internet of Things) device such as a TV, a speaker, a lamp, an air conditioner, a humidifier, a tongue, a window, and a blind.

Further, the object 300 according to the embodiment of the present invention is not provided with a function capable of communicating with the object control support system 200, another object 300, or an external system (not shown), or a memory means or a microprocessor is used. It may be a general object that has a predetermined physical position in the real world without being provided. For example, common objects such as thermometers, photographs, pictures, dolls, mirrors can be adopted as the object 300 according to an embodiment of the present invention.

On the other hand, according to one embodiment of the present invention, the object 300, which is a device, may further include an application program for performing the function according to the present invention. Such an application may exist in the form of a program module within the device. The character of such a program module may be generally similar to the spatial information management unit 210 and the object control management unit 220 of the object control support system 200 as described above. Here, at least a part of the application may be replaced with a hardware device or a firmware device capable of performing substantially the same or equivalent functions as necessary.

Configuration of object control support system

In the following, the internal configuration of the object control support system 200 and the functions of each component that perform important functions for the realization of the present invention will be described in detail.

FIG. 2 is a drawing schematically showing an internal configuration of an object control support system according to an embodiment of the present invention.

Referring to FIG. 2, the object control support system 200 according to the embodiment of the present invention can include a spatial information management unit 210, an object control management unit 220, a database 230, a communication unit 240, and a control unit 250. According to one embodiment of the present invention, the spatial information management unit 210, the object control management unit 220, the database 230, the communication unit 240, and the control unit 250 are program modules in which at least a part thereof communicates with an external system (not shown). Can be. Such a program module may be included in the object control support system 200 in the form of an operating system, an application module and other program modules, and may be physically stored on a variety of known storage devices. Further, such a program module may be stored in a remote storage device capable of communicating with the control support system 200 of the object. On the other hand, such a program module includes routines, subroutines, programs, objects, components, data structures, etc. that perform specific tasks described later or execute specific types of abstract data according to the present invention. Not limited to.

On the other hand, although the object control support system 200 has been described as described above, such an explanation is exemplary, and at least a part of the components or functions of the object control support system 200 is required. It may be embodied within an object 300 (eg, an IoT device) or an external system (not shown) to be controlled accordingly, or may be included within an object 300 or an external system (not shown). It is obvious to those skilled in the art. In some cases, all functions and components of the object control support system 200 are all executed within the object 300 or the external system (not shown) or all within the object 300 or the external system (not shown). May be included.

First, according to one embodiment of the present invention, the spatial information management unit 210 controls various objects existing in the surrounding space (hereinafter referred to as “target space”) by performing a predetermined gesture or operation by the user. It is possible to perform the function of acquiring the spatial information necessary for the purpose.

Specifically, the spatial information management unit 210 according to the embodiment of the present invention can acquire information necessary for identifying at least one user and an object existing in the target space. For example, information on the user's identifier, superior eye, etc. and information on the object's identifier, attributes, operating state, communication means, etc. can be acquired.

In addition, the spatial information management unit 210 according to the embodiment of the present invention can acquire information necessary for specifying the positions and postures of at least one user and an object existing in the target space. For example, a user (more specifically, the user's dominant eye and a body part that makes the user appear to overlap the object) specified in a real world coordinate system defined with respect to the target space. ) And information about the position and orientation of the object 300 identified in the real world coordinate system can be obtained.

On the other hand, according to one embodiment of the present invention, the spatial information acquired as described above is updated in real time with a predetermined time as a cycle, or an event such as a change in the position or posture of the user or the object 300. Can be dynamically updated by the occurrence of.

FIG. 3 is a drawing illustrating an exemplary situation in which various objects that can be controlled in the target space exist according to an embodiment of the present invention.

Referring to FIG. 3, the spatial information management unit 210 according to the embodiment of the present invention makes a gesture or operation for the user to identify an object existing in the target space 350 or to control the operation or function of the object. In doing so, a 0-dimensional point, a 1-dimensional line, a 2-dimensional region, or a 3-dimensional space that can be used as a reference can be set with reference to the attribute of the corresponding object. For example, reference points 301A to 307A can be set for objects such as lights 301, 306, speakers 304, 305, mirror 302, humidifier 303, and air conditioner 307 that have the attribute of pressing a button or turning a dial. Reference lines 308A to 310A can be set for objects such as blind 308, tongue 309, and window 310 that have the attribute of being slid up and down or left and right, and TV311 that has the attribute of being able to operate the two-dimensional screen. A two-dimensional reference area 311A may be set for an object such as, and a three-dimensional reference space 312A may be set for an object such as an augmented reality object 312 having an attribute to which an operation is applied with reference to a three-dimensional space.

On the other hand, the spatial information management unit 210 according to the embodiment of the present invention can define the object reference coordinate system with reference to the attributes of the object. For example, for an object such as a lighting 301, 306, a speaker 304, 305, a mirror 302, a humidifier 303, an air conditioner 307, which has an attribute to which an operation of pressing a button or turning a dial can be added, the reference points 301A to 307A of the object are set. A zero-dimensional object reference coordinate system as the origin can be defined. As another example, for an object such as a blind 308, a tongue 309, or a window 310 having an attribute of being slid up / down or left / right, a one-dimensional object reference centered on the reference lines 308A to 310A of the object. A coordinate system can be defined. As yet another example, for an object such as TV311 that has an attribute to which an operation for specifying a point, line, or region is applied to a two-dimensional screen, a two-dimensional object reference coordinate system corresponding to the two-dimensional reference region 311A of the object. Can be defined. As yet another example, for an object such as an augmented real object 312 having an attribute to which an operation is applied with reference to a predetermined 3D space, a 3D object reference coordinate system corresponding to the 3D reference space 312A of the object is defined. Can be done.

Next, the object control management unit 220 according to the embodiment of the present invention associates (that is, overlaps) with the target object that the user intends to control, such as a body part such as his / her fingertip, when the user sees from his / her own viewpoint. By performing an action (to make it visible), it is possible to identify the target object that one wants to control.

Further, the object control management unit 220 according to the embodiment of the present invention may perform a gesture matching the attributes of the target object by using a body part such as a fingertip that the user can see overlapping with the target object. , A remote controller, a laser pointer, or other vector control means can be used to control the movement or function of the target object by performing an operation that matches the attributes of the object.

Hereinafter, the operation of the object control management unit 220 according to the embodiment of the present invention will be described in detail.

First, according to one embodiment of the present invention, the object control management unit 220 can specify a control vector corresponding to a gesture performed by the user in a real-world coordinate system. Here, according to one embodiment of the present invention, when the user takes a gesture to make his / her fingertip appear to overlap with the object to be controlled when viewed from his / her own viewpoint, the gesture corresponds to the above-mentioned gesture. The control vector can be specified as a vector starting from the position of the user's eye (more accurately, the dominant eye) and ending at the position of the user's fingertip.

Further, according to one embodiment of the present invention, the object control management unit 220 can specify a control vector (vector) corresponding to an operation performed by the user in a coordinate system in the real world. Here, according to one embodiment of the present invention, when the user performs an operation of instructing an object to be controlled by using a vector control means such as a remote controller or a laser pointer, the gesture corresponds to the above-mentioned gesture. The control vector can be specified based on the attitude of the vector control means to be manipulated by the user or the direction of the control signal generated by the vector control means. For example, the control vector according to an embodiment of the present invention may be a vector connecting two points on the vector control means for specifying the posture of the vector control means, and is a vector corresponding to an axis defining the posture of the vector control means. It may be a vector corresponding to the direction of the instruction signal (laser signal of the laser pointer, infrared (IR) signal of the remote controller, etc.) generated by the vector control means, and the signal generation unit that generates the instruction signal by the vector control means. It may be a vector connecting a point corresponding to the position of the above and a predetermined reference point (which may be located on the opposite side of the signal generation unit) on the vector control means.

Then, according to one embodiment of the present invention, the object control management unit 220 uses the gesture of the user or the object indicated by the control vector specified in response to the gesture or operation of the user in the target space. It can be determined as the target object controlled by the operation.

Specifically, the object control management unit 220 according to the embodiment of the present invention refers to the spatial information about the user and the object, and determines the object indicated by the control vector in the target space as the target object. Can be done. More specifically, the object control management unit 220 according to the embodiment of the present invention can determine an object that intersects with or is close to the control vector or an extension of the control vector in the target space as the target object. In other words, according to one embodiment of the invention, the target object does not necessarily have to intersect the control vector or the extension of the control vector, between the control vector and the reference vector from the starting point of the control vector to the reference point of the object. Even when the angle of (that is, the error angle) is equal to or less than the preset level, the corresponding object can be determined to be the object (that is, the target object) indicated by the control vector.

Next, according to one embodiment of the present invention, the object control management unit 220 specifies mappings that specify control coordinates corresponding to the size or direction of the control vector in the target object reference coordinate system defined based on the attributes of the target object. (Mapping) can be performed.

Then, according to one embodiment of the present invention, the object control management unit 220 can determine the control command applied to the target object by referring to the value of the control coordinates specified by the mapping.

4 to 7 are drawings schematically showing a configuration for mapping a control vector of a real-world coordinate system and a control coordinate of a target object reference coordinate system according to an embodiment of the present invention.

First, referring to FIG. 4, it can be assumed that the operation or function of the air conditioner 307 is controlled by the gesture of the user. In such a case, the object control management unit 220 according to the embodiment of the present invention controls the position of the user's eye (that is, the superior eye) 410 as the starting point and the position of the user's fingertip 420 as the ending point. The air conditioner 307 indicated by the vector 430 (or an extension of the control vector 430) can be determined to be the target object controlled by the user's gesture. Here, according to an embodiment of the present invention, even if the control vector 430 does not accurately indicate the reference point 307A of the air conditioner 307, the control vector 430 is a reference from the start point 410 of the control vector 430 to the reference point 307A of the air conditioner 307. Within the permissible error range 460 (ie, error angle range) as compared to the vector (not shown), the air conditioner 307 can be determined to be the target object indicated by the control vector 430. ..

Subsequently, referring to FIG. 4, the object control management unit 220 according to the embodiment of the present invention is a control vector in the reference coordinate system of the 0-dimensional air conditioner defined based on the attributes of the air conditioner 307 which is the target object. A mapping can be performed that identifies the control coordinates corresponding to the size or direction of the 430. For example, the control coordinates are specified by the size of the control vector 430 (hereinafter referred to as “size”) and the rotation angle 470 (hereinafter referred to as “rotation angle”) of the control vector 430 centered on the reference point 307A of the air conditioner 307. Can be done.

Subsequently, with reference to FIG. 4, the object control management unit 220 according to the embodiment of the present invention can determine the control command applied to the air conditioner 307 with reference to the specified control coordinates. ..

For example, when the user sees from his / her own point of view, he / she makes a gesture of moving his / her fingertip in a direction close to the air conditioner 307 with his / her fingertip 420 corresponding to the reference point 307A of the air conditioner 307 in the coordinate system of the real world. When the control vector 430 is specified, the magnitude value of the control coordinates can be determined by mapping the control vector 430 to the control coordinates on the reference coordinate system of the air conditioner, and the magnitude value of the determined control coordinates can be determined. The air conditioner can be turned on or off based on.

As another example, when the user sees from his point of view, his fingertip 420 makes a gesture of rotating clockwise or counterclockwise around the reference point 307A of the air conditioner 307 to make a real-world coordinate system. When the control vector is specified in, the rotation angle value of the control coordinates can be determined by mapping the control vector 430 to the control coordinates on the reference coordinate system of the air conditioner, and the rotation angle value of the determined control coordinates can be determined. The set temperature or air volume of the air conditioner 307 can be adjusted based on the above.

As yet another example, when the user sees from his / her point of view, his / her fingertip 420 is in the right region of the reference point 307A of the air conditioner 307 (for example, the error angle is 3 to 6 degrees and the rotation angle is 45 to 135 degrees. Real-world coordinate system by performing a gesture to move the fingertip toward the right side of the reference point 307A of the air conditioner 307 (that is, a gesture to tab the right side area of the air conditioner 307 (Right Tab)) in a state corresponding to the area). When the control vector 430 is specified in, the size value and the rotation angle value of the control coordinates can be determined by mapping the control vector 430 to the control coordinates on the reference coordinate system of the air conditioner, and the determined control can be determined. The air volume of the air conditioner 307 can be adjusted based on the magnitude value and the rotation angle value of the coordinates.

As yet another example, when the user sees from his / her point of view, his / her fingertip 420 is in the lower region (for example, the error angle is 3 to 6 degrees and the rotation angle is 135 to 225) with respect to the reference point 307A of the air conditioner 307. A gesture to move from an upper area (eg, an area with an error angle of 3 to 6 degrees and a rotation angle of 315 to 45 degrees, etc.) (ie, a gesture to dispel the air conditioner 307 from bottom to top). When the control vector 430 is specified in the real-world coordinate system by performing Swipe)), the size value and rotation of the control coordinates are mapped by mapping the control vector 430 to the control coordinates on the reference coordinate system of the air conditioner. The angle value can be determined, and the wind direction of the air conditioner 307 can be adjusted based on the size value and the rotation angle value of the determined control coordinates.

As yet another example, when the user sees from his / her point of view, his / her fingertip 420 is in the right region (for example, the error angle is 3 to 6 degrees and the rotation angle is 135 to 225 degrees) with respect to the reference point 307A of the air conditioner 307. By performing a gesture to move from the central region (for example, the region where the error angle is 0 to 3 degrees) (that is, the gesture to tab the right side surface of the air conditioner 307 (Light-side Tab)). When the control vector 430 is specified in the world coordinate system, the size value and the angle of rotation value of the control coordinates can be determined by mapping the control vector 430 to the control coordinates on the reference coordinate system of the air conditioner. The operation mode of the air conditioner 307 may be changed based on the determined magnitude value and rotation angle value of the control coordinates.

Then, referring to FIG. 5, it can be assumed that the operation or function of the blind 308 is controlled by the gesture of the user. In such a case, the object control management unit 220 according to the embodiment of the present invention has the position of the user's eye (that is, the superior eye) 510 as the starting point and the position of the user's fingertips 521 and 522 as the ending point. The blind 308 indicated by the control vector 531 and 532 (or an extension of the control vector 531 and 532) can be determined to be the target object controlled by the user's gesture. Here, according to an embodiment of the present invention, even if the control vectors 531 and 532 do not accurately indicate the reference line 308A of the blind 308, the control vectors 531 and 532 are from the starting point 510 of the control vectors 531 and 532 to the blind 308. Within the permissible error range (ie, error angle range) relative to the reference vector (not shown) to the reference line 308A, the blind 308 is the target object indicated by the control vectors 531 and 532. Can be determined as a thing.

Subsequently, with reference to FIG. 5, the object control management unit 220 according to the embodiment of the present invention is a one-dimensional blind reference coordinate system defined based on the attributes of the blind 308 which is the target object, and is a control vector 531. Mapping can be performed to identify control coordinates corresponding to the size or direction of 532. For example, the control coordinates can be specified by the size of the control vectors 531 and 532 and the one-dimensional coordinates of the point where the control vectors 531 and 532 meet or are closest to the reference line 308A of the blind 308.

Subsequently, with reference to FIG. 5, the object control management unit 220 according to the embodiment of the present invention can determine the control command applied to the blind 308 with reference to the specified control coordinates. .. For example, when the user sees from his / her own point of view, the point where his / her fingertips 521 and 522 can be seen corresponding to the reference line 308A of the blind 308 is located relatively above the point 541 and relatively below. When the direction of the control vectors 531 and 532 changes in the real-world coordinate system by performing a gesture to move to the point 542, the control vectors 531 and 532 are mapped to the control coordinates on the blind reference coordinate system. By doing so, the one-dimensional coordinate value of the control coordinate can be changed, and the operation or function of raising or lowering the blind 308 can be performed based on the changing one-dimensional coordinate value.

Then, referring to FIG. 6, it can be assumed that the operation or function of the TV 311 is controlled by the gesture of the user. In such a case, the object control management unit 220 according to the embodiment of the present invention controls the position of the user's eye (that is, the superior eye) 610 as the starting point and the position of the user's fingertip 620 as the ending point. The TV 311 indicated by the vector 630 (or an extension of the control vector 630) can be determined to be the target object controlled by the user's gesture. Here, according to one embodiment of the present invention, even if the control vector 630 does not accurately indicate the reference region 311A of the TV311, the control vector 630 is a reference vector from the starting point 610 of the control vector 630 to the reference region 311A of the TV311. Within the permissible error range (ie, error angle range) as compared to (not shown), the TV 311 can be determined to be the target object indicated by the control vector 630.

Subsequently, with reference to FIG. 6, the object control management unit 220 according to the embodiment of the present invention is a two-dimensional TV reference coordinate system defined based on the attributes of the target object TV311 and is a control vector 630. A mapping can be performed that identifies the control coordinates corresponding to the size or direction. For example, the control coordinates can be specified by the size of the control vector 630 and the two-dimensional coordinates of the point where the control vector 630 meets or is closest to the reference region 311A of the TV 311.

Subsequently, with reference to FIG. 6, the object control management unit 220 according to the embodiment of the present invention can determine the control command applied to the TV 311 with reference to the specified control coordinates. For example, a real-world coordinate system by making a gesture that allows the user's fingertip 620 to correspond to a point 640 near the top left on the left side of the reference area 311A of the TV311 when viewed from his point of view. When the direction of the control vector 630 is specified in, the two-dimensional coordinate value of the control coordinate can be specified by mapping the control vector 630 to the control coordinate on the TV reference coordinate system, and the specified two-dimensional coordinate can be specified. An operation or function of selecting a graphic element displayed in a pixel corresponding to the specified two-dimensional coordinate value in the display screen of the TV 311 based on the value can be performed.

Then, with reference to FIG. 7, it can be assumed that the user's gesture controls the movement or function of the augmented reality object 312 (ie, the virtual object) with respect to the doll. Here, it should be clarified that visual information about the augmented reality object 312 can be provided to the user through devices such as smartphones, tablets, smart glasses, and projectors. In such a case, the object control management unit 220 according to the embodiment of the present invention starts from the position of the user's eye (that is, the superior eye) 710 and sets the position of the user's fingertips 721, 722, 723. The augmented reality object 312 indicated by the control vectors 731, 732, 733 (or an extension of the control vectors 731, 732, 733) as the end point is determined to be the target object controlled by the user's gesture. be able to. Here, according to one embodiment of the present invention, the control vectors 731, 732, 733 may be control vectors 731, 732, even if the control vectors 731, 732, 733 do not accurately indicate the reference space 312A of the augmented reality object 312. Controlling the augmented reality object 312 when it is within the permissible error range (ie, error angle range) compared to the reference vector (not shown) from the starting point 710 of 733 to the reference space 312A of the augmented reality object 312. It can be determined to be the target object indicated by the vectors 731, 732, 733.

Subsequently, with reference to FIG. 7, the object control management unit 220 according to the embodiment of the present invention is a three-dimensional expanded reality object reference coordinate system defined based on the attributes of the augmented reality object 312 which is the target object. , Control vectors 731, 732, 733 can perform mappings that specify control coordinates corresponding to the size or direction of the vector 731, 732, 733. For example, the control coordinates can be specified by the size of the control vectors 731, 732, 733 and the three-dimensional coordinates of the point where the control vectors 731, 732, 733 meet or are closest to the reference space 308A of the augmented reality object 312.

Subsequently, with reference to FIG. 7, the object control management unit 220 according to the embodiment of the present invention determines the control command applied to the augmented reality object 312 for the doll with reference to the specified control coordinates. can do. For example, when the user sees from his / her own point of view, the points where his / her fingertips 721, 722, and 723 can be seen corresponding to the reference space 312A of the expanded reality object 312 are the ear portion 741 and the face portion 742 of the expanded reality object 312, respectively. When the directions of the control vectors 731, 732, and 733 are specified in the real-world coordinate system by performing a gesture so as to be the foot portion 743, the control vectors 731, 732, and 733 are used in the extended reality object reference coordinate system. By mapping to the above control coordinates, the three-dimensional coordinate values of the control coordinates can be specified respectively, and the ear part 741, the face part 742 and the foot part of the augmented reality object 312 can be specified based on the three-dimensional coordinate values specified respectively. Each of the 743s may perform the selected action or function.

In FIGS. 4 to 7, the embodiment in which the object is controlled based on the gesture performed by the user has been mainly described, but the present invention is not necessarily limited to the examples exemplified in FIGS. 4 to 7. However, it should be clarified that any number of changes can be made within the range in which the object of the present invention can be achieved.

As another example, a control vector corresponding to an operation performed by a user using a vector control means such as a remote controller or a laser pointer is specified, a target object indicated by the specified control vector is determined, and the determination is made. There is also an embodiment in which mapping for specifying the control coordinates corresponding to the control vector is performed in the reference coordinate system of the target object to be performed, and the control command applied to the target object is determined with reference to the specified control coordinates. You can imagine as many as you want.

8 and 9 are drawings schematically showing a configuration in which a control vector is specified by an operation performed by a user using a vector control means according to an embodiment of the present invention.

Referring to FIGS. 8 and 9, points 820 and 920 corresponding to the position of the signal generation unit that generates a control signal (infrared signal of the remote controller 800 or laser signal of the laser pointer 900) and vector control by the vector control means 800 and 900. The vector connecting the predetermined reference points 810 and 910 located on the opposite side of the signal generation unit by the means 800 and 900 can be specified as the control vectors 830 and 930.

Subsequently, with reference to FIGS. 8 and 9, the air conditioner 307, which is the object indicated by the control vectors 830, 930 identified as described above or the extension lines 840, 940 of the control vector, is controlled by the operation of the user. Can be determined as a target object.

On the other hand, according to one embodiment of the present invention, the database 230 can store various spatial information about the user and the object existing in the space where the control of the object according to the present invention is performed, and can control the object. A variety of information about control vectors, control coordinates and control instructions derived in performing can be stored. For example, in FIG. 2, the database 230 is illustrated as being included in the object control support system 200, but the database 230 is separate from the object control support system 200 as needed by those skilled in the art who embody the present invention. It may be configured in. On the other hand, the database 230 in the present invention is a concept including a computer-readable recording medium, and may be a database in a broad sense including not only a database in a narrow sense but also data recording based on a file system, and is merely a set of logs. Even if there is, the database 230 in the present invention can be used as long as it can be searched and data can be extracted.

Next, the communication unit 240 according to the embodiment of the present invention performs a function of enabling the object control support system 200 to communicate with an external device.

Finally, the control unit 250 according to the embodiment of the present invention performs a function of controlling the flow of data between the spatial information management unit 210, the object control management unit 220, the database 230, and the communication unit 240. That is, the control unit 250 controls the flow of data from the outside or between each component of the object control support system 200, so that the spatial information management unit 210, the object control management unit 220, the database 230, and the communication unit 240, respectively. Control to perform unique functions.

The embodiments according to the present invention described above can be embodied in the form of program instructions that can be carried out through various computer components and recorded on a non-transient computer-readable recording medium. The non-transient computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the non-transient computer-readable recording medium are specially designed and configured for the present invention or are known to those skilled in the art of computer software. It may be usable. Examples of non-transient computer readable recording media include hard disks, magnetic media such as floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and floptic discs. Includes such magnetic-optical media, and hardware devices specially configured to store and execute program commands such as ROMs, RAMs, flash memories, and the like. Examples of program instructions include not only machine language code as created by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform the processing according to the invention and vice versa.

The invention has been described above by means of specific matters such as specific components and limited examples and drawings, which are provided to aid in a more general understanding of the invention. However, the present invention is not limited to the above-described embodiment, and a person having ordinary knowledge in the technical field to which the present invention belongs should be able to make various modifications and modifications from such a description.

Therefore, the idea of the present invention should not be limited to the above-mentioned examples, and not only the scope of claims described later but also all that are equally or equivalently modified from the scope of claims. Can be said to belong to the category of the idea of the present invention.

100: User device 200: Object control support system 210: Spatial information management unit 220: Object control management unit 230: Database 240: Communication unit 250: Control unit

Claims (7)

It ’s a way to help control the object,
The stage of identifying the control vector corresponding to the gesture or operation performed by the user in the real-world coordinate system,
A step of determining an object designated by the control vector as a target object controlled by a gesture or operation performed by the user.
For the target object with reference to the stage of performing mapping to specify the control coordinates corresponding to the size or direction of the control vector in the target object reference coordinate system, and the control coordinates specified by the mapping. Including the step of determining the control command to be applied
The target object reference coordinate system is a 0-dimensional object reference coordinate system defined by a reference point, a one-dimensional object reference coordinate system defined by a reference line, and a two-dimensional object reference coordinate system defined by a reference region. And the reference point, the reference line, the reference area, and the reference, depending on the attributes associated with the gesture or operation that can be applied to the object, including the three-dimensional object reference coordinate system defined by the reference space. The 0-dimensional object reference coordinate system, the 1-dimensional object reference coordinate system, by the reference point, the reference line, the reference region, or the reference space in which any one of the spaces is selectively set and set. Any one of the two-dimensional target object reference coordinate system and the three-dimensional target object reference coordinate system is selectively defined for the target object .
The method in which the control coordinates are specified based on the relationship between the control vector and the reference point, the reference line, the reference region or the reference space. The method according to claim 1, wherein the control vector is a vector starting from the position of the eyes of the user and ending at the position of another body part of the user. The method according to claim 1, wherein the control vector is a vector specified based on the posture of the vector control means to be operated or the direction of the control signal generated by the vector control means. The method according to claim 1, wherein the object is a real object or a virtual object existing around the user. In the target object determination stage,
1. The object that intersects with or is closest to the control vector or an extension of the control vector in the real-world coordinate system is determined as the target object with reference to spatial information about the user and the object. The method described in. A non-transient computer-readable recording medium recording a computer program for carrying out the method according to claim 1. It is a system to support the control of the object.
A control vector corresponding to a gesture or operation performed by the user is specified in a real-world coordinate system, and an object indicated by the control vector is determined as a target object controlled by the gesture or operation performed by the user. Then, in the target object reference coordinate system, mapping is performed to specify the control coordinates corresponding to the size or direction of the control vector, and the control coordinates specified by the mapping are referred to with respect to the target object. The object control management unit that determines the control command to be applied,
Including a spatial information management unit that stores spatial information about the user and the object.
The target object reference coordinate system is a 0-dimensional object reference coordinate system defined by a reference point, a one-dimensional object reference coordinate system defined by a reference line, and a two-dimensional object reference coordinate system defined by a reference region. And the reference point, the reference line, the reference area, and the reference, depending on the attributes associated with the gesture or operation that can be applied to the object, including the three-dimensional object reference coordinate system defined by the reference space. The 0-dimensional object reference coordinate system, the 1-dimensional object reference coordinate system, by the reference point, the reference line, the reference region, or the reference space in which any one of the spaces is selectively set and set. Any one of the two-dimensional target object reference coordinate system and the three-dimensional target object reference coordinate system is selectively defined for the target object .
The control coordinates are specified based on the relationship between the control vector and the reference point, the reference line, the reference region or the reference space.

Images