+

WO2018198499A1 - Dispositif de traitement d'informations, procédé de traitement d'informations et support d'enregistrement - Google Patents

Dispositif de traitement d'informations, procédé de traitement d'informations et support d'enregistrement Download PDF

Info

Publication number
WO2018198499A1
WO2018198499A1 PCT/JP2018/006020 JP2018006020W WO2018198499A1 WO 2018198499 A1 WO2018198499 A1 WO 2018198499A1 JP 2018006020 W JP2018006020 W JP 2018006020W WO 2018198499 A1 WO2018198499 A1 WO 2018198499A1
Authority
WO
WIPO (PCT)
Prior art keywords
imaging unit
unit
information processing
input
processing apparatus
Prior art date
Application number
PCT/JP2018/006020
Other languages
English (en)
Japanese (ja)
Inventor
友久 田中
Original Assignee
ソニー株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ソニー株式会社 filed Critical ソニー株式会社
Priority to JP2019515117A priority Critical patent/JPWO2018198499A1/ja
Priority to US16/495,588 priority patent/US20200042105A1/en
Priority to EP18791869.3A priority patent/EP3617851B1/fr
Publication of WO2018198499A1 publication Critical patent/WO2018198499A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/0304—Detection arrangements using opto-electronic means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0487—Interaction techniques based on graphical user interfaces [GUI] using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser

Definitions

  • the present disclosure relates to an information processing apparatus, an information processing method, and a recording medium.
  • so-called information processing devices In recent years, with the advancement of communication technology and the miniaturization of various devices, the type of equipment called so-called information processing devices has also diversified, not limited to PCs (Personal Computers) etc., like smartphones and tablet terminals, Information processing apparatuses configured to be carried by users are also becoming popular.
  • a so-called wearable device has also been proposed that is configured to be used while being carried by a user wearing it on a part of the body.
  • a wearable device a device (hereinafter referred to as a “head-mounted device”) used by being worn on the head, such as an HMD (Head Mounted Display) or glasses-type wearable device.
  • HMD Head Mounted Display
  • glasses-type wearable device For example.
  • input devices such as buttons, switches, touch sensors, and the like are generally known as examples of an input interface for a user to input various types of information to the information processing apparatus.
  • an input interface for a user to input various types of information to the information processing apparatus.
  • a head-mounted device it may be difficult for a user to directly view an input device provided in a part of the housing due to the characteristic of being mounted on the head and used. Usability is poor compared to the case where the input interface is visible.
  • gesture input may be adopted as an input interface for inputting various information to the information processing apparatus without using an input device such as a button or a switch.
  • gesture input requires processing with a relatively high load such as image recognition, the power consumption tends to increase.
  • an information processing apparatus capable of recognizing a user's operation input in a more preferable manner without using an input device provided in a housing of the apparatus. suggest.
  • the information processing apparatus includes: a determination unit that determines whether or not the imaging unit is in a predetermined shielding state; and a recognition unit that recognizes a user operation input according to the predetermined shielding state. Is provided.
  • the computer includes determining whether the imaging unit is in a predetermined shielding state, and recognizing a user operation input according to the predetermined shielding state. An information processing method is provided.
  • the computer is caused to determine whether or not the imaging unit is in a predetermined shielding state and to recognize a user's operation input according to the predetermined shielding state.
  • a recording medium on which the program is recorded is provided.
  • FIG. 3 is an explanatory diagram for describing an example of a schematic configuration of an input / output device according to the embodiment. It is explanatory drawing for demonstrating the outline
  • FIG. 10 is an explanatory diagram for describing an example of a user interface according to Modification 1.
  • FIG. 10 is an explanatory diagram for describing an example of a user interface according to Modification 2.
  • FIG. 3 is a functional block diagram illustrating a configuration example of a hardware configuration of an information processing apparatus configuring an information processing system according to an embodiment of the present disclosure.
  • FIG. 1 is an explanatory diagram for explaining an example of a schematic configuration of an information processing system according to an embodiment of the present disclosure, and applies various contents to a user by applying a so-called AR (Augmented Reality) technique. An example in the case of presenting is shown.
  • AR Augmented Reality
  • reference numeral m111 schematically shows an object (for example, a real object) located in the real space.
  • Reference numerals v131 and v133 schematically indicate virtual contents (for example, virtual objects) presented so as to be superimposed on the real space. That is, the information processing system 1 according to the present embodiment superimposes a virtual object on a real space object such as the real object m111 and presents it to the user based on, for example, the AR technology.
  • FIG. 1 in order to make the characteristics of the information processing system according to the present embodiment easier to understand, both real objects and virtual objects are presented together.
  • the information processing system 1 includes an information processing device 10 and an input / output device 20.
  • the information processing apparatus 10 and the input / output apparatus 20 are configured to be able to transmit and receive information to and from each other via a predetermined network.
  • the type of network connecting the information processing apparatus 10 and the input / output apparatus 20 is not particularly limited.
  • the network may be a so-called wireless network such as a network based on the Wi-Fi (registered trademark) standard.
  • the network may be configured by the Internet, a dedicated line, a LAN (Local Area Network), a WAN (Wide Area Network), or the like.
  • the network may include a plurality of networks, and at least a part of the network may be configured as a wired network.
  • the input / output device 20 has a configuration for acquiring various input information and presenting various output information to a user holding the input / output device 20.
  • the presentation of output information by the input / output device 20 is controlled by the information processing device 10 based on the input information acquired by the input / output device 20.
  • the input / output device 20 acquires information for recognizing the real object m111 (for example, a captured real space image) as input information, and outputs the acquired information to the information processing device 10.
  • the information processing device 10 recognizes the position of the real object m111 in the real space based on the information acquired from the input / output device 20, and causes the input / output device 20 to present the virtual objects v131 and v133 based on the recognition result.
  • the input / output device 20 can present the virtual objects v131 and v133 to the user so that the virtual objects v131 and v133 are superimposed on the real object m111 based on so-called AR technology.
  • the input / output device 20 is configured as a so-called head-mounted device that is used by a user wearing at least a part of the head, and may be configured to detect the user's line of sight.
  • the information processing apparatus 10 for example, based on the detection result of the user's line of sight by the input / output device 20, is a target desired by the user (for example, the real object m111, the virtual objects v131 and v133, etc.). May be specified as an operation target.
  • the information processing apparatus 10 may specify a target to which the user's line of sight is directed as an operation target with a predetermined operation on the input / output device 20 as a trigger.
  • the information processing apparatus 10 may provide various services to the user via the input / output device 20 by specifying the operation target and executing the process associated with the operation target.
  • the information processing apparatus 10 uses, as the user's operation input, movement of at least a part of the user's body (for example, change in position and orientation, gesture, etc.) based on the input information acquired by the input / output device 20. It may be recognized and various processes may be executed according to the recognition result of the operation input. As a specific example, the input / output device 20 acquires information for recognizing a user's hand (for example, an image of a captured hand) as input information, and outputs the acquired information to the information processing device 10.
  • a user's hand for example, an image of a captured hand
  • the information processing device 10 recognizes a hand movement (for example, a gesture) based on the information acquired from the input / output device 20, and gives an instruction from the user (that is, an operation input by the user) according to the recognition result of the movement. Recognize And the information processing apparatus 10 may control the display (for example, the display position and attitude
  • “user operation input” may be regarded as an input corresponding to an instruction from the user, that is, an input reflecting the user's intention.
  • “user operation input” may be simply referred to as “user input”.
  • the input / output device 20 and the information processing device 10 are illustrated as different devices, but the input / output device 20 and the information processing device 10 may be integrally configured. Details of the configuration and processing of the input / output device 20 and the information processing device 10 will be described later.
  • FIG. 2 is an explanatory diagram for explaining an example of a schematic configuration of the input / output device according to the present embodiment.
  • the input / output device 20 is configured as a so-called head-mounted device that is used by a user wearing at least a part of the head.
  • the input / output device 20 is configured as a so-called eyewear type (glasses type) device, and at least one of the lenses 293a and 293b is a transmission type display (display unit 211). It is configured as.
  • the input / output device 20 includes imaging units 201a and 201b, an operation unit 207, and a holding unit 291 corresponding to a frame of glasses. Further, the input / output device 20 may include imaging units 203a and 203b.
  • the input / output device 20 includes the imaging units 203a and 203b.
  • the holding unit 291 causes the display unit 211, the imaging units 201a and 201b, the imaging units 203a and 203b, and the operation unit 207 to move to the user's head. So as to have a predetermined positional relationship with respect to the portion.
  • the input / output device 20 may include a sound collection unit for collecting the user's voice.
  • the lens 293a corresponds to a right eye side lens
  • the lens 293b corresponds to a left eye side lens. That is, the holding unit 291 holds the display unit 211 so that the display unit 211 (in other words, the lenses 293a and 293b) is positioned in front of the user's eyes when the input / output device 20 is mounted.
  • the imaging units 201a and 201b are configured as so-called stereo cameras, and when the input / output device 20 is attached to the user's head, the imaging units 201a and 201b face the direction in which the user's head is facing (that is, the front of the user). In this way, the holding unit 291 holds each of them. At this time, the imaging unit 201a is held near the right eye of the user, and the imaging unit 201b is held near the left eye of the user. Based on such a configuration, the imaging units 201a and 201b capture a subject located in front of the input / output device 20 (in other words, a real object located in real space) from different positions.
  • the input / output device 20 acquires an image of a subject located in front of the user, and based on the parallax between images captured by the imaging units 201a and 201b, the input / output device 20 (and thus the user's). It is possible to calculate the distance from the viewpoint) to the subject.
  • the configuration and method are not particularly limited as long as the distance between the input / output device 20 and the subject can be measured.
  • the distance between the input / output device 20 and the subject may be measured based on a method such as multi-camera stereo, moving parallax, TOF (Time Of Flight), or Structured Light.
  • TOF refers to projecting light such as infrared rays to a subject and measuring the time until the posted light is reflected and returned by the subject for each pixel, and based on the measurement result, This is a method of obtaining an image (so-called distance image) including the distance (depth) of the image.
  • the Structured Light is a distance image that includes the distance (depth) to the subject based on the change in the pattern obtained from the imaging result by irradiating the subject with light such as infrared rays and imaging it. It is a method to obtain.
  • the moving parallax is a method of measuring the distance to the subject based on the parallax even in a so-called monocular camera. Specifically, the subject is imaged from different viewpoints by moving the camera, and the distance to the subject is measured based on the parallax between the captured images. At this time, the distance to the subject can be measured with higher accuracy by recognizing the moving distance and moving direction of the camera using various sensors. Note that the configuration of the imaging unit (for example, a monocular camera, a stereo camera, or the like) may be changed according to the distance measurement method.
  • the imaging units 203a and 203b are respectively held by the holding unit 291 so that when the input / output device 20 is mounted on the user's head, the user's eyeball is positioned within each imaging range.
  • the imaging unit 203a is held such that the user's right eye is positioned within the imaging range. Based on such a configuration, the line of sight of the right eye is directed based on the image of the right eyeball imaged by the imaging unit 203a and the positional relationship between the imaging unit 203a and the right eye. It becomes possible to recognize the direction.
  • the imaging unit 203b is held so that the left eye of the user is positioned within the imaging range.
  • the input / output device 20 includes a configuration including both the imaging units 203a and 203b. However, only one of the imaging units 203a and 203b may be provided.
  • the operation unit 207 is configured to accept an operation from the user with respect to the input / output device 20.
  • the operation unit 207 may be configured by an input device such as a touch panel and buttons.
  • the operation unit 207 is held at a predetermined position of the input / output device 20 by the holding unit 291. For example, in the example illustrated in FIG. 2, the operation unit 207 is held at a position corresponding to a temple of glasses.
  • the input / output device 20 is provided with, for example, an acceleration sensor or an angular velocity sensor (gyro sensor), and the movement of the head of the user wearing the input / output device 20 (in other words, the input / output device). 20 itself) can be detected.
  • the input / output device 20 detects components of the yaw direction, the pitch direction, and the roll direction as movements of the user's head. A change in at least one of the position and posture of the head may be recognized.
  • the input / output device 20 can recognize a change in its position and posture in the real space according to the movement of the user's head.
  • the input / output device 20 displays the content on the display unit 211 so that the virtual content (that is, the virtual object) is superimposed on the real object located in the real space. It can also be presented.
  • the input / output device 20 may estimate its own position and orientation in real space (that is, self-position) based on a technique called SLAM (simultaneous localization and mapping), for example. The estimation result may be used for presenting the virtual object.
  • SLAM simultaneous localization and mapping
  • SLAM is a technology that performs self-position estimation and creation of an environment map in parallel by using an imaging unit such as a camera, various sensors, an encoder, and the like.
  • SLAM particularly Visual SLAM
  • the position and orientation of the imaging unit are estimated as information indicating a relative change based on the detection result of the sensor by providing the input / output device 20 with various sensors such as an acceleration sensor and an angular velocity sensor, for example. Is possible.
  • the method is not necessarily limited to a method based on detection results of various sensors such as an acceleration sensor and an angular velocity sensor.
  • HMD Head Mounted Display
  • examples of a head-mounted display device include a see-through HMD, a video see-through HMD, and a retinal projection HMD.
  • the see-through type HMD uses, for example, a half mirror or a transparent light guide plate to hold a virtual image optical system including a transparent light guide unit or the like in front of the user's eyes and display an image inside the virtual image optical system. Therefore, a user wearing a see-through HMD can view an external scenery while viewing an image displayed inside the virtual image optical system.
  • the see-through type HMD for example, is based on the AR technology, based on the recognition result of at least one of the position and orientation of the see-through type HMD, with respect to the optical image of the real object located in the real space. It is also possible to superimpose virtual object images.
  • a specific example of the see-through HMD is a so-called glasses-type wearable device in which a portion corresponding to a lens of glasses is configured as a virtual image optical system.
  • the input / output device 20 illustrated in FIG. 2 corresponds to an example of a see-through HMD.
  • the video see-through HMD When the video see-through HMD is mounted on the user's head or face, the video see-through HMD is mounted so as to cover the user's eyes, and a display unit such as a display is held in front of the user's eyes. Further, the video see-through HMD has an imaging unit for imaging a surrounding landscape, and displays an image of a landscape in front of the user captured by the imaging unit on the display unit. With such a configuration, it is difficult for a user wearing a video see-through HMD to directly view an external landscape, but it is possible to confirm the external landscape from an image displayed on the display unit. Become. At this time, the video see-through HMD superimposes a virtual object on an external landscape image according to the recognition result of at least one of the position and orientation of the video see-through HMD, for example, based on the AR technology. You may let them.
  • a projection unit is held in front of the user's eyes, and the image is projected from the projection unit toward the user's eyes so that the image is superimposed on an external landscape. More specifically, in the retinal projection type HMD, an image is directly projected from the projection unit onto the retina of the user's eye, and the image is formed on the retina. With such a configuration, it is possible to view a clearer video even for a myopic or farsighted user. In addition, the user wearing the retinal projection type HMD can view an external landscape while viewing an image projected from the projection unit.
  • the retinal projection type HMD is based on, for example, the AR technology, and an optical image of a real object located in the real space according to at least one of the positions and orientations of the retinal projection type HMD. It is also possible to superimpose a virtual object image on the image.
  • the configuration of the input / output device 20 according to the present embodiment has been described on the assumption that the AR technology is applied.
  • the configuration of the input / output device 20 is not necessarily limited.
  • the input / output device 20 according to the present embodiment may be configured as an HMD called an immersive HMD.
  • the immersive HMD is mounted so as to cover the user's eyes, and a display unit such as a display is held in front of the user's eyes. For this reason, it is difficult for a user wearing an immersive HMD to directly view an external landscape (that is, a real world landscape), and only the image displayed on the display unit enters the view.
  • the immersive HMD can give an immersive feeling to the user who is viewing the image.
  • An example of an input interface for a user to input various types of information to the information processing apparatus includes input devices such as buttons, switches, and touch sensors. Also in a head-mounted device such as the input / output device 20 described with reference to FIG. 2, an input device such as a button or a touch sensor (for example, the operation unit 207 illustrated in FIG. 2) In some cases (for example, a part of a holding unit that holds a display unit, an imaging unit, or the like).
  • the housing vibrates due to an operation on the input interface, and the vibration is held by the housing. May be transmitted to the display unit or imaging unit.
  • the relative positional relationship between the user's eyes and the display unit and the imaging unit changes, and the real object and the virtual object presented to be superimposed on the real object are: The user may not be visually recognized due to the correct positional relationship.
  • gesture input may be adopted as an input interface for inputting various information to the information processing apparatus without using an input device such as a button or a switch.
  • gesture input for example, by analyzing an image captured by an imaging unit or the like, a gesture using a part such as a hand is recognized, and a user input is recognized according to the recognition result of the gesture. Accordingly, the user can input information to the information processing apparatus by a more intuitive operation such as a gesture without operating the input device provided on the housing (that is, the input device that is difficult to view). It becomes possible to input.
  • gesture input requires processing with a relatively high load such as image recognition
  • power consumption tends to increase.
  • head-mounted devices such as those described with reference to FIG. 2 are often battery-powered due to the characteristics of being worn on the head and carried like smartphones. Sometimes used. In such a device, it is more desirable that the power consumption be reduced.
  • an example of a technique capable of recognizing a user input without using an input device provided in a housing of the apparatus and further reducing a processing load related to the recognition Propose about.
  • FIGS. 3 and 4 are explanatory diagrams for explaining the outline of the input interface according to the present embodiment.
  • the information processing apparatus 10 is, for example, an imaging unit that captures an image of an external environment (for example, real object recognition, self-position estimation, etc.) such as a stereo camera provided in a head-mounted device.
  • the imaging unit used in the above is used for recognizing user input. Therefore, in this description, an outline of the input interface according to the present embodiment will be described by taking as an example the case where the imaging units 201a and 201b are used for user input recognition in the input / output device 20 described with reference to FIG. To do.
  • the user can give various instructions to the information processing apparatus 10 by covering at least a part of the imaging units 201a and 201b with a part such as a hand. .
  • the information processing apparatus 10 recognizes a user input according to whether or not at least some of the imaging units 201a and 201b are in a predetermined shielding state.
  • the predetermined shielding state includes, for example, a state where substantially the entire angle of view of a desired imaging unit is shielded.
  • the predetermined shielding state is described as indicating a state in which substantially the entire angle of view of the desired imaging unit is shielded, but the present invention is not necessarily limited to this state.
  • FIG. 3 shows a situation where the angle of view of the imaging unit 201a is shielded by the user's hand U11.
  • the information processing apparatus 10 determines whether substantially the entire angle of view of the imaging unit 201a is shielded based on a predetermined method. It is recognized that a predetermined input has been made (that is, a user input is recognized).
  • the imaging unit 201a corresponds to an example of a “first imaging unit”. That is, the above-described determination regarding the shielding state of the imaging unit 201a (for example, determination of whether substantially the entire angle of view of the imaging unit 201a is blocked) corresponds to an example of “first determination”.
  • FIG. 4 shows a situation where the angle of view of the imaging unit 201b is blocked by the user's hand U13.
  • the information processing apparatus 10 determines whether substantially the entire angle of view of the imaging unit 201 b is shielded, and according to the determination result. Recognize user input.
  • the imaging unit 201b corresponds to an example of a “second imaging unit”. That is, the above determination regarding the shielding state of the imaging unit 201b corresponds to an example of “second determination”.
  • the method is not particularly limited as long as it is possible to determine whether substantially the entire angle of view of each of the imaging units 201a and 201b is blocked.
  • the information processing apparatus 10 determines whether substantially the entire angle of view of each of the imaging units 201a and 201b is shielded based on the brightness of the images captured by the imaging units 201a and 201b. You may judge.
  • a method for determining whether substantially the entire angle of view of the image capturing unit is shielded according to the brightness of the image captured by the predetermined image capturing unit will be described later in detail as an example.
  • various sensors such as a proximity sensor and a distance measuring sensor.
  • the distance detection result is When the value is equal to or less than the threshold value, it may be determined that substantially the entire angle of view is shielded.
  • the information processing apparatus 10 can recognize a user input depending on, for example, whether substantially the entire angle of view of any of the imaging units 201a and 201b is shielded. It becomes.
  • the information processing apparatus 10 may recognize a user input according to a combination of imaging units in which the entire angle of view is shielded among the imaging units 201a and 201b.
  • the information processing apparatus 10 has a case where substantially the entire angle of view is shielded only for one of the imaging units 201a and 201b when both of the imaging units 201a and 201b are shielded. It is also possible to recognize that an input different from that is made.
  • FIG. 5 is a block diagram illustrating an example of a functional configuration of the information processing system 1 according to the present embodiment. Therefore, hereinafter, as described with reference to FIG. 1, assuming that the information processing system 1 includes the information processing apparatus 10 and the input / output apparatus 20, the configurations of the information processing apparatus 10 and the input / output apparatus 20 are described. This will be described in more detail. As illustrated in FIG. 5, the information processing system 1 may include a storage unit 190.
  • the input / output device 20 includes imaging units 201 a and 201 b and an output unit 210.
  • the output unit 210 includes a display unit 211.
  • the output unit 210 may include a sound output unit 213.
  • the imaging units 201a and 201b correspond to the imaging units 201a and 201b described with reference to FIG. Note that the imaging units 201a and 201b may be simply referred to as “imaging unit 201” if they are not particularly distinguished.
  • the display unit 211 corresponds to the display unit 211 described with reference to FIG.
  • the acoustic output unit 213 includes an acoustic device such as a speaker and outputs sound and sound corresponding to information to be output.
  • the information processing apparatus 10 includes a determination unit 101, a recognition unit 103, a process execution unit 105, and an output control unit 107.
  • the determination unit 101 acquires information according to the imaging result of the image from the imaging unit 201, and substantially the entire angle of view of the imaging unit is blocked by some real object (for example, a user's hand) according to the acquired information. It is determined whether or not it has been done.
  • the determination unit 101 acquires an image captured by the image capturing unit 201 from the image capturing unit 201, and displays the image of the image capturing unit 201 according to the brightness of the acquired image (for example, luminance distribution for each pixel). It may be determined whether substantially the entire corner is shielded. As a more specific example, the determination unit 101 calculates the average value of the luminance of each pixel of the acquired image, and when the calculated average value of luminance is equal to or less than the threshold, the image of the imaging unit 201 that has captured the image. It may be determined that substantially the entire corner is shielded.
  • the determination unit 101 calculates the average value of the luminance of each pixel of the acquired image, and when the calculated average value of luminance is equal to or less than the threshold, the image of the imaging unit 201 that has captured the image. It may be determined that substantially the entire corner is shielded.
  • the determination unit 101 acquires an image captured from the image capturing unit 201 and determines that recognition of an object in real space (that is, a real object) is difficult based on the acquired image, It may be determined that substantially the entire angle of view of the imaging unit 201 is shielded.
  • the determination unit 101 has difficulty extracting feature points for recognizing a real object from an acquired image (for example, when the number of extracted feature points is equal to or less than a threshold), the image It may be determined that substantially the entire angle of view of the imaging unit 201 that captured the image is shielded.
  • the above-described example is merely an example, and the method is not particularly limited as long as the determination unit 101 can determine whether substantially the entire angle of view of the imaging unit 201 is blocked.
  • the determination unit 101 detects the proximity of a real object to the imaging unit 201 by a distance measuring sensor, a proximity sensor, or the like, the entire angle of view of the imaging unit 201 is shielded. You may judge.
  • the number of imaging units 201 to be determined by the determination unit 101 is not particularly limited. As a specific example, the determination unit 101 may determine only one of the imaging units 201a and 201b, or may determine both the imaging units 201a and 201b. In addition, the determination unit 101 may set other image capturing units other than the image capturing units 201a and 201b as a determination target. That is, the determination unit 101 may determine three or more imaging units as a determination target.
  • the timing at which the determination unit 101 performs the above-described determination is not particularly limited.
  • the determination unit 101 may periodically perform the determination at every predetermined timing.
  • the determination unit 101 may perform the above determination according to a predetermined trigger.
  • the determination unit 101 may perform the above determination when predetermined display information such as an operation menu for prompting user input is displayed on the display unit 211. In this case, for example, the determination unit 101 may recognize whether or not predetermined display information is displayed on the display unit 211 based on a notification from the output control unit 107 described later.
  • the determination unit 101 notifies the recognition unit 103 of information indicating a determination result as to whether or not substantially the entire angle of view of the imaging unit 201 is blocked. At this time, for example, when the determination unit 101 determines that substantially the entire angle of view of the predetermined imaging unit 201 is blocked, the determination unit 101 may notify the recognition unit 103. In addition, when there are a plurality of candidates for the imaging unit 201 to be determined, the determination unit 101 may notify the recognition unit 103 of information indicating a determination result for each imaging unit 201.
  • the recognition unit 103 acquires information indicating a determination result as to whether or not substantially the entire angle of view of the imaging unit 201 is blocked from the determination unit 101, and recognizes user input based on the acquired information. At this time, the recognizing unit 103 may recognize the user input in accordance with information related to user input recognition displayed on the display unit 211 and information indicating the determination result.
  • FIG. 6 is an explanatory diagram for explaining an example of the input interface according to the present embodiment, and shows an example of an operation menu presented via the display unit 211 of the input / output device 20.
  • reference numeral V101 schematically shows an optical image of real space visually recognized by the user.
  • Reference numeral V ⁇ b> 103 indicates an area (that is, a drawing area) where display information (for example, a virtual object) is presented via the display unit 211.
  • Reference numerals V105 and V107 indicate examples of display information presented as operation menus. Specifically, the display information V105 is associated with an operation menu that indicates permission of execution of a predetermined process, and the display information V107 is associated with an operation menu that indicates cancellation of execution of the process. Yes.
  • the recognition unit 103 for example, of the imaging unit 201b positioned on the left side relative to the user wearing the input / output device 20 (that is, the imaging unit 201b shown in FIG. 2).
  • the recognition unit 103 recognizes that the user has given an instruction to affirm the execution of the predetermined process. That is, the recognizing unit 103 recognizes the above operation by the user as a user input meaning affirmation.
  • the recognition unit 103 is configured to block substantially the entire angle of view of the imaging unit 201a (that is, the imaging unit 201a illustrated in FIG. 2) positioned on the right side relative to the user wearing the input / output device 20. Recognizes that the operation menu corresponding to the display information V107 has been selected. In this case, the recognizing unit 103 recognizes that an instruction to cancel execution of a predetermined process has been given by the user. That is, the recognizing unit 103 recognizes the above-described operation by the user as a user input meaning cancellation.
  • the recognition unit 103 may execute the above-described processing related to recognition of user input in response to a predetermined trigger.
  • the recognition unit 103 may execute processing related to user input recognition when predetermined display information such as an operation menu for prompting user input is displayed on the display unit 211. .
  • the recognition unit 103 may recognize whether or not predetermined display information is displayed on the display unit 211 based on a notification from the output control unit 107, for example.
  • the recognition unit 103 outputs information indicating the recognition result of the user input to the process execution unit 105.
  • the process execution unit 105 is configured to execute various functions (for example, applications) provided by the information processing apparatus 10 (and thus the information processing system 1). For example, the process execution unit 105 extracts a corresponding application from a predetermined storage unit (for example, a storage unit 190 described later) in accordance with the recognition result of the user input by the recognition unit 103, and executes the extracted application. Good. Further, the process execution unit 105 may control the operation of the application being executed according to the recognition result of the user input by the recognition unit 103. For example, the process execution unit 105 may switch the subsequent operation of the application being executed according to the operation menu selected by the user. Further, the process execution unit 105 may output information indicating execution results of various applications to the output control unit 107.
  • a predetermined storage unit for example, a storage unit 190 described later
  • the output control unit 107 presents the information to the user by causing the output unit 210 to output various types of information to be output.
  • the output control unit 107 may present the display information to the user by causing the display unit 211 to display display information to be output.
  • the output control part 107 may show the said information to a user by making the sound output part 213 output the sound according to the information used as output object.
  • the output control unit 107 may acquire information indicating the execution results of various applications from the process execution unit 105, and present output information corresponding to the acquired information to the user via the output unit 210.
  • the output control unit 107 displays display information corresponding to the operation menu of the application, such as the display information V105 and V107 shown in FIG. It may be displayed.
  • the output control unit 107 may cause the display unit 211 to display display information indicating the execution result of a desired application.
  • the output control unit 107 may cause the acoustic output unit 213 to output output information corresponding to the execution result of a desired application as voice or sound.
  • the output control unit 107 may notify the determination unit 101 and the recognition unit 103 of information indicating the output status of various types of output information via the output unit 210.
  • the output control unit 107 indicates that the information is displayed.
  • the determination unit 101 and the recognition unit 103 may be notified.
  • the storage unit 190 is a storage area for storing various data temporarily or permanently.
  • the storage unit 190 may store data for the information processing apparatus 10 to execute various functions.
  • the storage unit 190 may store data (for example, a library) for executing various applications, management data for managing various settings, and the like.
  • the functional configuration of the information processing system 1 shown in FIG. 5 is merely an example, and the functional configuration of the information processing system 1 is not necessarily the example shown in FIG. 5 if the processing of each configuration described above can be realized. It is not limited to only.
  • the input / output device 20 and the information processing device 10 may be integrally configured.
  • the storage unit 190 may be included in the information processing apparatus 10 or a recording medium external to the information processing apparatus 10 (for example, externally attached to the information processing apparatus 10). Recording medium).
  • a part of the configuration of the information processing apparatus 10 may be provided outside the information processing apparatus 10 (for example, a server).
  • FIG. 7 is a flowchart showing an example of a flow of a series of processes of the information processing system 1 according to the present embodiment.
  • the information processing apparatus 10 acquires information according to the imaging result of an image from a predetermined imaging unit 201 held in the input / output device 20, and displays an image of the imaging unit according to the acquired information. It is determined whether or not substantially the entire corner is blocked by some real object (for example, a user's hand) (S101).
  • the information processing apparatus 10 determines that the angle of view is blocked.
  • the user input is recognized according to the imaging unit (S105).
  • the information processing apparatus 10 executes processing according to the recognition result of the user input (S107).
  • the information processing apparatus 10 may execute a corresponding application according to the recognition result of the user input.
  • the information processing apparatus 10 may present output information corresponding to the execution result of the application to the user via the output unit 210.
  • the information processing apparatus 10 does not execute the processing according to the reference numerals S103 and 107, You may change to the subsequent processing.
  • the trigger for the information processing apparatus 10 to execute a series of processes indicated by reference numerals S101 to S107 is not particularly limited.
  • the information processing apparatus 10 may execute the series of processes in response to a predetermined trigger.
  • the information processing apparatus 10 may execute the above-described series of processes when the user is prompted to input information via the input / output device 20.
  • FIG. 8 shows an example of an image picked up by the image pickup unit when the angle of view of the predetermined image pickup unit is blocked by a hand, and shows a case where the distance between the image pickup unit and the hand is about 20 cm. ing.
  • FIG. 9 is a graph showing the luminance distribution of each pixel of the image shown in FIG.
  • the horizontal axis indicates the luminance of the pixel
  • the vertical axis indicates the frequency.
  • the luminance of each pixel indicates a value from 0 to 255, and the higher the value, the higher the luminance.
  • FIG. 9 in the example shown in FIG. 8, it can be seen that many pixels having relatively high luminance are distributed.
  • leakage of light from the external environment from an area not shielded by the hand contributes. It is presumed that
  • FIG. 10 shows an example of an image picked up by the image pickup unit when the angle of view of the predetermined image pickup unit is shielded by hand, and shows a case where the distance between the image pickup unit and the hand is about 10 cm. ing.
  • the area shielded by the hand in the angle of view of the imaging unit is wider, and the brightness of the entire image is also darker.
  • FIG. 11 is a graph showing the luminance distribution of each pixel of the image shown in FIG. Note that the horizontal and vertical axes in FIG. 11 are the same as those in the graph shown in FIG.
  • the image shown in FIG. 10 has more pixels with lower luminance than the image shown in FIG. That is, it can be seen that the brightness of the entire image shown in FIG. 10 is darker than the brightness of the entire image shown in FIG.
  • FIG. 12 shows an example of an image captured by the imaging unit when the angle of view of the predetermined imaging unit is shielded by a hand, and shows a case where the distance between the imaging unit and the hand is about 1 cm. ing.
  • FIG. 13 is a graph showing the luminance distribution of each pixel of the image shown in FIG. Note that the horizontal and vertical axes in FIG. 13 are the same as those in the graph shown in FIG.
  • the image shown in FIG. 12 has more pixels with lower luminance than the image shown in FIG.
  • each pixel has a brightness that is brighter than black, although it is slight, because light from the external environment leaks from the gap between the imaging unit and the hand. I guess that.
  • FIG. 14 shows an example of an image captured by the imaging unit when the angle of view of the predetermined imaging unit is shielded by a hand, and shows a case where the distance between the imaging unit and the hand is about 1 mm. ing.
  • FIG. 15 is a graph showing the luminance distribution of each pixel of the image shown in FIG. Note that the horizontal and vertical axes in FIG. 15 are the same as those in the graph shown in FIG. As can be seen by comparing FIG. 15 and FIG. 13, the image shown in FIG.
  • the imaging unit used in the present embodiment when the luminance distribution of each pixel of the captured image is as shown in FIG. It can be regarded as a boundary (threshold value) for determining whether or not substantially the entire corner is shielded. That is, in the imaging unit used in the present embodiment, for example, when the average value of the luminance of each pixel of the captured image shows a value of 77 or less, substantially the entire angle of view of the imaging unit is shielded. Can be considered as being.
  • the threshold for determining whether substantially the entire angle of view of the imaging unit is blocked is the configuration, installation position, installation method, and the like of the imaging unit. Needless to say, it may be appropriately changed according to the various conditions.
  • Modification 1 Notification of information regarding the shielding state of the angle of view of the imaging unit
  • the head-mounted device such as the input / output device 20 is used by being mounted on the head, so that the user is positioned in front of each of the components of the input / output device 20 according to the wearing state. It becomes difficult to visually recognize other parts other than the part directly. Therefore, for example, when the imaging units 201a and 201b illustrated in FIG. 2 are used for determination of user input, the user can directly visually recognize the imaging units 201a and 201b in a state where the input / output device 20 is mounted. It can be difficult.
  • the information processing apparatus 10 outputs the notification information according to the shielding state of the angle of view of the imaging unit used for the determination of the user input, so that the shielding state is determined by the user. May be notified.
  • FIG. 17 is an explanatory diagram for describing an example of a user interface according to the first modification.
  • an example of a user interface will be described assuming that the imaging units 201a and 201b are used for determination of user input, assuming that the input / output device 20 illustrated in FIG. 2 is used.
  • the objects to which reference numerals V201 to V207 are attached correspond to the objects to which reference numerals V101 to v107 are attached in the example described with reference to FIG. Omitted.
  • images taken by the imaging units 201a and 201b used for determination of user input are displayed in the drawing area V203.
  • images captured by the imaging units 201 a and 201 b are presented to the user via the display unit 211.
  • a reference symbol V213 indicates an image captured by the imaging unit 201b that is positioned on the left side relative to the user wearing the input / output device 20, and the region indicated by the reference symbol V209 Is displayed.
  • the imaging unit 201b is associated with an operation menu corresponding to the display information V205. Based on such a situation, for example, when the user selects an operation menu corresponding to the display information V205, the angle of view of the imaging unit 201b is blocked by checking the image V213 displayed in the region V209. It is possible to visually confirm the situation (that is, whether or not substantially the entire angle of view is blocked). For example, in the example illustrated in FIG. 17, the angle of view of the imaging unit 201b is blocked by the user's hand indicated by reference symbol U13, and the user's hand U13 is captured as an object in the image V213.
  • reference numeral V215 indicates an image captured by the imaging unit 201a located on the right side relative to the user wearing the input / output device 20, and is displayed in the area indicated by reference numeral V211.
  • the imaging unit 201a is associated with an operation menu corresponding to the display information V207. Based on such a situation, for example, when the user selects an operation menu corresponding to the display information V207, the angle of view of the imaging unit 201a is blocked by checking the image V215 displayed in the region V211. It is possible to visually confirm the situation (that is, whether or not substantially the entire angle of view is blocked).
  • the imaging unit is confirmed while confirming the image presented via the display unit 211. It is possible to shield the angle of view with a hand or the like.
  • the example described with reference to FIG. 17 is merely an example, and if the user can be notified of the situation where the angle of view of the imaging unit used for determination of user input is blocked, the user is notified.
  • the type of information that is, notification information
  • the notification method of the information and the like.
  • the information processing apparatus 10 uses the notification information according to the situation where the angle of view of the imaging unit used for determination of the user input is shielded (for example, the ratio of being shielded) as sound, such as a speaker. You may show to a user via a sound output part. As a specific example, when the angle of view of the imaging unit located on the left side relative to the user is shielded, the information processing apparatus 10 starts from a speaker located on the left side relative to the user. Sounds such as sound effects may be output at a volume corresponding to the ratio at which the angle of view is shielded.
  • the information processing apparatus 10 is output from a speaker, for example, as the user's hand approaches a predetermined imaging unit (that is, the brightness of an image captured by the imaging unit becomes darker). You may control so that the volume of the sound to be increased may become larger.
  • Modification 1 an example of a user interface in the case of notifying the user of the situation where the angle of view of the imaging unit is blocked will be described with reference to FIG.
  • Modification 2 Presentation of information on the method of shielding the angle of view of the imaging unit
  • the user directly places other portions of the input / output device 20 other than the portion positioned in front of the eyes according to the wearing state. It becomes difficult to visually recognize. Therefore, it may be difficult for the user to directly view the imaging units (for example, the imaging units 201a and 201b illustrated in FIG. 2) used for determining the user input in a state where the input / output device 20 is mounted. .
  • the imaging units for example, the imaging units 201a and 201b illustrated in FIG. 2 used for determining the user input in a state where the input / output device 20 is mounted.
  • the information processing apparatus 10 provides notification information for notifying a method of shielding the angle of view of the miscellaneous portion used for determination of user input (that is, an operation method). You may notify a user.
  • FIG. 18 is an explanatory diagram for describing an example of a user interface according to the second modification.
  • an example of a user interface will be described assuming that the imaging units 201a and 201b are used for determination of user input, assuming that the input / output device 20 illustrated in FIG. 2 is used.
  • the objects to which reference numerals V301 to V307 are attached correspond to the objects to which reference numerals V101 to v107 are attached in the example described with reference to FIG. Omitted.
  • the information processing apparatus 10 uses the operation method (that is, the method of shielding the angle of view of the imaging unit) when a state in which no user input is detected continues for a predetermined period after prompting the user to perform an operation. ) To present notification information.
  • the operation method that is, the method of shielding the angle of view of the imaging unit
  • the information processing apparatus 10 prompts the user to perform an operation by presenting the notification information V309 in the drawing area V303.
  • the information processing apparatus 10 notifies the user of the operation method by presenting the notification information V311 and V313 when a state in which no user input is detected continues for a predetermined period or longer. Yes.
  • the notification information V311 is an image of a method of shielding the angle of view of the imaging unit 201b located on the left side relative to the user as an operation method for selecting an operation menu corresponding to the display information V305. As shown. That is, the information processing apparatus 10 notifies the user of an operation method for selecting an operation menu corresponding to the display information V305 by presenting the notification information V311 in the vicinity of the display information V305.
  • the notification information V313 shows, as an image, a method of shielding the angle of view of the imaging unit 201a positioned on the right side relative to the user as an operation method for selecting an operation menu corresponding to the display information V307. Yes. That is, the information processing apparatus 10 notifies the user of an operation method for selecting an operation menu corresponding to the display information V307 by presenting the notification information V313 in the vicinity of the display information V307.
  • the example illustrated in FIG. 18 is merely an example, and if the operation method (that is, the method of shielding the angle of view of the imaging unit) can be notified to the user, the type of notification information and the notification method are not necessarily limited. It is not limited to the example shown in FIG.
  • the information processing apparatus 10 recognizes a user input according to whether or not substantially the entire angle of view of a predetermined imaging unit is blocked.
  • the information processing apparatus 10 according to the modification 3 has the first shielding state and the shielding amount of the angle of view as compared to the first shielding state as the shielding state of the angle of view of the predetermined imaging unit.
  • a small second shielding state is identified and each is recognized as a different user input.
  • examples of the first shielding state include a state where substantially the entire angle of view of a predetermined imaging unit is shielded.
  • a 2nd shielding state the state by which only a part of angle of view of the said imaging part is shielded is mentioned.
  • the information processing apparatus 10 identifies a state where substantially the entire angle of view of the predetermined imaging unit is shielded and a state where only a part of the angle of view is shielded. explain.
  • a state where substantially the entire angle of view of a predetermined imaging unit is blocked is associated with a state where a predetermined button is pressed, only a part of the angle of view is blocked. May be associated with a state in which the button is half-pressed.
  • the criteria for determining each of the state where the angle of view is not occluded, the state where only a part of the angle of view is occluded, and the state where almost the entire angle of view is occluded are not particularly limited. What is necessary is just to set suitably according to a utilization form. As a specific example, when the user input is determined based on the brightness of the image captured by the imaging unit, only a part of the angle of view is blocked, and substantially the entire angle of view is blocked. What is necessary is just to set suitably the threshold value for discriminating each of the present state.
  • Modification 3 an example of a method for recognizing user input in accordance with the shielding state of the angle of view of the imaging unit is described.
  • Modification 4 An example of control according to the external environment
  • the brightness of the image May vary depending on the brightness of the external environment.
  • “brightness of the external environment” may be regarded as the intensity of ambient light around the information processing apparatus 10 in a state where the angle of view of the imaging unit is not shielded. Therefore, for example, depending on whether the external environment is bright or dark, the manner in which the brightness of an image to be captured varies depending on whether the angle of view of the imaging unit is blocked or not. For example, in a situation where the external environment is relatively bright, the amount of change in the brightness of the captured image is relatively large depending on whether substantially the entire angle of view of the imaging unit is blocked.
  • an image of an image to be captured is determined depending on whether or not substantially the entire angle of view of the imaging unit is shielded. The amount of change in brightness decreases.
  • the information processing apparatus 10 separately detects the brightness of the external environment using, for example, an illuminance sensor and the like.
  • a threshold value for determining whether or not the whole is shielded may be dynamically controlled.
  • the information processing apparatus 10 determines whether or not the angle of view of the imaging unit is blocked when the brightness detection result of the external environment is equal to or less than the threshold (that is, user input). Recognition) may be temporarily suppressed.
  • the method is not necessarily limited to a method using an illuminance sensor as long as recognition of user input can be temporarily suppressed depending on whether the external environment is bright or not.
  • the external environment is bright
  • the brightness of the image captured by the other imaging unit It becomes brighter.
  • the image was picked up by another image pickup unit even when substantially the entire angle of view of only some of the image pickup units is blocked. The image becomes darker. From such characteristics, for example, when the information processing apparatus 10 determines that substantially the entire angle of view of only a part of the plurality of imaging units is blocked (in other words, approximately the entire angle of view).
  • the user input may be recognized according to the shielding state only when the number of imaging units determined to be shielded is equal to or less than the threshold.
  • the information processing apparatus 10 is configured such that when the number of imaging units that are determined to be substantially shielded for the entire angle of view exceeds the threshold value (as a result, substantially the entire angle of view is shielded for all of the plurality of imaging units).
  • the user input may be restricted.
  • Modification 5 Example of user input recognition method using a plurality of imaging units
  • the angle of view of each of the two image pickup units can be blocked by using both hands.
  • the information processing apparatus 10 may recognize a user input according to a combination of imaging units in which substantially the entire angle of view is shielded among the plurality of imaging units.
  • a combination of imaging units in which substantially the entire angle of view is shielded among the plurality of imaging units.
  • four imaging units are used for user input recognition and the angle of view of each imaging unit is shielded by hand, at most two of the four imaging units are used. The angle of view of the imaging unit is shielded.
  • each of six states ( 4 C 2 ) corresponding to a combination of two imaging units whose field angles are shielded among the four imaging units, and any one of the four imaging units.
  • Each of the four states ( 4 C 1 ) corresponding to the case where the angle of view of only one imaging unit is blocked can be individually recognized as different user inputs.
  • the information processing apparatus 10 may recognize a user input according to a combination of imaging units in which substantially the entire angle of view is shielded among a plurality of imaging units.
  • a function that requires an explicit instruction from the user such as a shutdown
  • a combination of imaging units that cannot easily block the angle of view unless both hands are used May be assigned.
  • the function assigned to the above operation is not limited to shutdown.
  • a function (so-called undo) for canceling a previously executed process may be assigned to the above operation.
  • the information processing apparatus 10 determines in time division whether substantially the entire angle of view of any of the plurality of imaging units is shielded in a predetermined time width.
  • the user input may be recognized according to the imaging unit that is entirely shielded and the timing at which the shielding is determined.
  • the information processing apparatus 10 recognizes different user inputs according to the order in which the angles of view are shielded in time division. May be. That is, the information processing apparatus 10 performs a process in which each of the field angles is shielded in the order of the imaging unit 201a and the imaging unit 201b, and in a case where each field angle is shielded in the order of the imaging unit 201b and the imaging unit 201a. It may be recognized as a different user input.
  • an image of the imaging unit 201a positioned on the right side relative to the user after substantially the entire angle of view of the imaging unit 201b positioned on the left side relative to the user is shielded. It is assumed that substantially the entire corner is shielded. In this case, for example, the information processing apparatus 10 has been operated with directionality from the left side to the right side according to the timing at which substantially the entire angle of view of each of the imaging units 201b and 201a is blocked. You may recognize.
  • the entire angle of view of the imaging unit 201b positioned relatively on the left side of the user is It shall be shielded.
  • the information processing apparatus 10 has been operated with directionality from the right side to the left side according to the timing at which substantially the entire angle of view of each of the imaging units 201a and 201b is blocked. You may recognize.
  • the information processing apparatus 10 can also recognize an operation having directionality such as a so-called swipe operation.
  • the information processing apparatus 10 may recognize different user inputs depending on the direction in which an imaging unit in which substantially the entire angle of view is shielded among a plurality of imaging units is installed. For example, when the input interface according to the present embodiment is applied to a device such as a smartphone, when the entire angle of view of the imaging unit on the front side is shielded, the device is placed upside down. It may be recognized and locked.
  • Modification 6 An example of a method for recognizing user input in accordance with an imaging state
  • a sixth modification as an example of a user input recognition method using an imaging unit, a case where the imaging state of an image by the imaging unit is used for user input recognition will be described.
  • various states relating to image capture may differ from the case where the angle of view is not shielded.
  • focus control for example, AF: Autofocus
  • the information processing apparatus 10 may determine that substantially the entire angle of view of the imaging unit is shielded.
  • the information processing apparatus 10 determines whether or not substantially the entire angle of view of the imaging unit is shielded based on the exposure control (AE: Automatic Exposure) state or the like (that is, recognition of user input). You may use it.
  • AE Automatic Exposure
  • the information processing apparatus 10 determines whether substantially the entire angle of view of a predetermined imaging unit used for user input recognition is blocked. Recognize user input.
  • the situation where the angle of view of the imaging unit is shielded is not necessarily the case where the user is intentionally shielded using a hand or the like.
  • a certain object for example, another person other than the user
  • crosses in front of the imaging unit to temporarily block the angle of view of the imaging unit.
  • a certain object for example, another person other than the user
  • the information processing apparatus 10 determines that substantially the entire field angle of the predetermined imaging unit is shielded, the state in which the field angle is shielded is the intention of the user. It may be possible to prevent erroneous recognition of user input by verifying whether or not it is caused by an operation.
  • the information processing apparatus 10 determines whether substantially the entire angle of view of the imaging unit is blocked based on an image captured by a predetermined imaging unit, the information processing apparatus 10 Depending on the mode of change, it may be verified whether or not the angle of view is blocked by a user's intentional operation.
  • the image change for example, brightness
  • the change in the image may be regarded as a change in the shielding amount of the angle of view.
  • the information processing apparatus 10 determines that at least a part of the angle of view of the predetermined imaging unit is shielded, images captured by the imaging unit before and after the determination It may be verified whether or not the angle of view is blocked by the user's intentional operation according to the change rate (the change rate of the shielding amount). That is, whether or not the operation input is recognized is controlled according to the change speed of the shielding amount of the angle of view of the imaging unit.
  • the change speed is equal to or higher than a predetermined value
  • recognition of a user's operation input may be restricted when change rate is below a predetermined value.
  • the information processing apparatus 10 may set a determination time for determining whether substantially the entire angle of view of the predetermined imaging unit is shielded. That is, the information processing apparatus 10 may control whether to recognize a user input according to the duration of the predetermined shielding state. More specifically, the information processing apparatus 10 is configured such that a state in which substantially the entire angle of view of a predetermined imaging unit is shielded continues for the determination time or longer (that is, the duration is equal to or longer than the determination time). In other words, it may be recognized that the angle of view is blocked by the user's intentional operation.
  • the information processing apparatus 10 may control the determination time according to a combination of imaging units in which substantially the entire angle of view is blocked. .
  • the information processing apparatus 10 may control the determination time to be relatively short.
  • a situation in which the angle of view of only one of the plurality of imaging units is shielded does not always occur due to a user's intentional operation, but occurs accidentally due to an event. There is also a possibility to do. Therefore, in this case, the information processing apparatus 10 may perform control so that the determination time is longer than when the angle of view of the plurality of imaging units is shielded.
  • the information processing apparatus 10 is provided between an imaging unit and an object (for example, a hand) that blocks an angle of view of the imaging unit using various sensors such as a distance measurement sensor and a proximity sensor. Using the distance detection result, it may be verified whether or not the angle of view is blocked by the user's intentional operation.
  • the distance between the imaging unit and the shielding object is several centimeters.
  • the distance between the imaging unit and the shielding object is at least a few tens of centimeters apart It is estimated to be.
  • the information processing apparatus 10 determines that the angle of view of the predetermined imaging unit is shielded, the detection result of the distance between the imaging unit and the shielding object is equal to or less than the threshold value. In this case, it may be recognized that the angle of view is blocked by a user's intentional operation.
  • the information processing apparatus 10 temporarily suppresses recognition of user input according to the brightness of the external environment by an illuminance sensor or the like, so that the external environment is dark. The occurrence of a situation that erroneously recognizes that the angle of view of the imaging unit is shielded may be prevented.
  • Modification 8 Example of control combined with other input interface
  • modification 8 an example of control when the input interface according to the present embodiment is combined with another input interface will be described.
  • the modification 8 by using the recognition result of the user input based on the input interface according to the present embodiment as a function similar to the shift key in the keyboard input, the user input via the other input interface is performed. A case where the recognition result is controlled will be described.
  • the number of recognizable user inputs is determined according to the number of keys from the characteristic of recognizing user inputs according to pressed keys.
  • the keyboard input it is possible to selectively switch the user input recognized according to the pressed key depending on whether or not the shift key is pressed. With such a mechanism, it is possible to recognize user inputs having a pattern larger than the number of keys in keyboard input.
  • the information processing apparatus 10 recognizes a user input that is subsequently input when it is determined that substantially the entire angle of view of the predetermined imaging unit is blocked, for example. It may be recognized that the result switching is instructed.
  • the information processing apparatus 10 recognizes the gesture as an operation for selecting a target virtual object when a gesture of simply hitting the virtual object is performed.
  • the information processing apparatus 10 uses the gesture as a target virtual object. You may recognize as operation for erasing.
  • the correspondence relationship between a predetermined gesture and a user input recognized by the gesture may be selectively switched according to a combination of imaging units in which substantially the entire angle of view is blocked.
  • the image capturing units 201a and 201b illustrated in FIG. 2 are used for user input, the image capturing unit 201a and 201b continues depending on which of the angle of view is almost entirely shielded. Different user inputs may be recognized based on the gesture input.
  • substantially the entire angle of view of both the imaging units 201a and 201b is shielded
  • substantially the entire angle of view of only one of the imaging units 201a and 201b is shielded based on the continuously input gesture.
  • a user input different from the case where it is performed may be recognized.
  • the input interface according to the present embodiment can also be used as a trigger for detecting a predetermined operation via another input interface.
  • the input interface according to the present embodiment can also be used as a trigger for detecting a predetermined operation via another input interface.
  • the number of recognizable patterns can be limited. Therefore, by combining the input interface according to the present embodiment with gesture input, for example, it is possible to expect effects such as improvement of gesture recognition accuracy and reduction of processing load related to gesture recognition.
  • the information processing apparatus 10 when the information processing apparatus 10 controls the recognition result of the user input via another input interface based on the input interface according to the present embodiment, the information processing apparatus 10 notifies the user of the control status. May be.
  • the information processing apparatus 10 switches the correspondence between a gesture input thereafter and a user input because substantially the entire angle of view of a predetermined imaging unit is shielded.
  • the display information indicating that the switching is being performed may be fed back to the user via the display unit 211.
  • the user can recognize that the correspondence relationship between the gesture input thereafter and the user input recognized by the gesture is switched.
  • Modification 9 Example of user input recognition method using sound collection unit
  • a function equivalent to the input interface according to the present embodiment is realized by using a device different from the imaging unit.
  • a sound collection unit such as a microphone
  • the information processing apparatus 10 receives the user input according to the sound collection result of the sound (in other words, acoustic noise) generated when the sound collection unit is struck by the sound collection unit used for user input recognition. May be recognized.
  • the operation of shielding substantially the entire angle of view of the predetermined imaging unit is replaced with the operation of hitting the predetermined sound collecting unit.
  • the user input can be recognized in the same manner as the information processing system 1 according to the embodiment.
  • notification information for example, a graph indicating the sound collection result of the sound
  • the user may be notified via the display unit 211.
  • notification information indicating the position of the sound collection unit may be notified to the user via the display unit 211.
  • the angle of view is spread over the entire periphery of the camera. From such a situation, for example, when the omnidirectional camera is used for determination of user input, the information processing apparatus 10 divides the angle of view of the omnidirectional camera into a plurality of partial areas, and A part of the partial area may be used for determination of user input. That is, the information processing apparatus 10 determines whether or not substantially all of a predetermined partial area is shielded from the angle of view of the omnidirectional camera, or a combination of partial areas where substantially the entire partial area is shielded. The user input may be recognized according to the above. In this case, the information processing apparatus 10 may notify the user of notification information for notifying the user of an area used for determination of user input via the display unit 211.
  • FIG. 19 is a functional block diagram illustrating a configuration example of the hardware configuration of the information processing apparatus configuring the information processing system according to an embodiment of the present disclosure.
  • the information processing apparatus 900 constituting the information processing system according to the present embodiment mainly includes a CPU 901, a ROM 902, and a RAM 903.
  • the information processing apparatus 900 further includes a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, and a connection port 923. And a communication device 925.
  • the CPU 901 functions as an arithmetic processing unit and a control unit, and controls all or a part of the operation in the information processing apparatus 900 according to various programs recorded in the ROM 902, the RAM 903, the storage apparatus 919, or the removable recording medium 927.
  • the ROM 902 stores programs used by the CPU 901, calculation parameters, and the like.
  • the RAM 903 primarily stores programs used by the CPU 901, parameters that change as appropriate during execution of the programs, and the like. These are connected to each other by a host bus 907 constituted by an internal bus such as a CPU bus.
  • a host bus 907 constituted by an internal bus such as a CPU bus.
  • the host bus 907 is connected to an external bus 911 such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 909.
  • an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925 are connected to the external bus 911 via an interface 913.
  • the input device 915 is an operation means operated by the user, such as a mouse, a keyboard, a touch panel, a button, a switch, a lever, and a pedal. Further, the input device 915 may be, for example, remote control means (so-called remote control) using infrared rays or other radio waves, or an external connection device such as a mobile phone or a PDA corresponding to the operation of the information processing device 900. 929 may be used. Furthermore, the input device 915 includes an input control circuit that generates an input signal based on information input by a user using the above-described operation means and outputs the input signal to the CPU 901, for example. A user of the information processing apparatus 900 can input various data and instruct a processing operation to the information processing apparatus 900 by operating the input device 915. For example, the input unit 221 illustrated in FIG. 7 can be configured by the input device 915.
  • the output device 917 is a device that can notify the user of the acquired information visually or audibly. Examples of such devices include CRT display devices, liquid crystal display devices, plasma display devices, EL display devices, display devices such as lamps, audio output devices such as speakers and headphones, printer devices, and the like.
  • the output device 917 outputs results obtained by various processes performed by the information processing apparatus 900. Specifically, the display device displays results obtained by various processes performed by the information processing device 900 as text or images.
  • the audio output device converts an audio signal composed of reproduced audio data, acoustic data, and the like into an analog signal and outputs the analog signal. For example, the output unit 210 illustrated in FIG.
  • the storage device 919 is a data storage device configured as an example of a storage unit of the information processing device 900.
  • the storage device 919 includes, for example, a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device.
  • the storage device 919 stores programs executed by the CPU 901 and various data.
  • the drive 921 is a reader / writer for a recording medium, and is built in or externally attached to the information processing apparatus 900.
  • the drive 921 reads information recorded on a removable recording medium 927 such as a mounted magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs the information to the RAM 903.
  • the drive 921 can also write a record to a removable recording medium 927 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory that is mounted.
  • the removable recording medium 927 is, for example, a DVD medium, an HD-DVD medium, a Blu-ray (registered trademark) medium, or the like.
  • the removable recording medium 927 may be a compact flash (registered trademark) (CF: CompactFlash), a flash memory, an SD memory card (Secure Digital memory card), or the like. Further, the removable recording medium 927 may be, for example, an IC card (Integrated Circuit card) on which a non-contact IC chip is mounted, an electronic device, or the like.
  • CF CompactFlash
  • SD memory card Secure Digital memory card
  • the connection port 923 is a port for directly connecting to the information processing apparatus 900.
  • Examples of the connection port 923 include a USB (Universal Serial Bus) port, an IEEE 1394 port, a SCSI (Small Computer System Interface) port, and the like.
  • As another example of the connection port 923 there are an RS-232C port, an optical audio terminal, an HDMI (registered trademark) (High-Definition Multimedia Interface) port, and the like.
  • the communication device 925 is a communication interface configured with, for example, a communication device for connecting to a communication network (network) 931.
  • the communication device 925 is, for example, a communication card for wired or wireless LAN (Local Area Network), Bluetooth (registered trademark), or WUSB (Wireless USB).
  • the communication device 925 may be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), a modem for various communication, or the like.
  • the communication device 925 can transmit and receive signals and the like according to a predetermined protocol such as TCP / IP, for example, with the Internet or other communication devices.
  • the communication network 931 connected to the communication device 925 is configured by a wired or wireless network, and may be, for example, the Internet, a home LAN, infrared communication, radio wave communication, satellite communication, or the like. .
  • FIG. 19 an example of a hardware configuration capable of realizing the functions of the information processing apparatus 900 that configures the information processing system according to the embodiment of the present disclosure has been shown.
  • Each component described above may be configured using a general-purpose member, or may be configured by hardware specialized for the function of each component. Therefore, it is possible to change the hardware configuration to be used as appropriate according to the technical level at the time of carrying out this embodiment.
  • FIG. 19 various configurations corresponding to the information processing apparatus 900 configuring the information processing system are naturally provided.
  • a computer program for realizing each function of the information processing apparatus 900 constituting the information processing system according to the present embodiment as described above can be produced and mounted on a personal computer or the like.
  • a computer-readable recording medium storing such a computer program can be provided.
  • the recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like.
  • the above computer program may be distributed via a network, for example, without using a recording medium.
  • the number of computers that execute the computer program is not particularly limited.
  • the computer program may be executed by a plurality of computers (for example, a plurality of servers) in cooperation with each other.
  • the information processing apparatus 10 determines whether substantially the entire angle of view of a predetermined imaging unit used for recognition of user input is blocked. The user input is recognized according to the determination result.
  • an input device i.e., an input device provided on the casing of the device
  • the method is not limited as long as it can be recognized that substantially the entire angle of view of the predetermined imaging unit is shielded. As a specific example, it is also possible to determine whether substantially the entire angle of view of the imaging unit is blocked based on the brightness of the image captured by the imaging unit.
  • the input interface according to the present embodiment the user input recognition does not require a relatively heavy processing such as image recognition for realizing gesture input. It is also possible to suppress power. Therefore, for example, the input interface according to the present embodiment has high affinity with a so-called head-mounted device (particularly, a portable device) such as the input / output device 20 described with reference to FIG.
  • the input interface according to the present embodiment has been described focusing on the case where the input interface is applied to a head-mounted device as illustrated in FIG. 2, but the application target of the input interface is not necessarily limited. .
  • the input interface according to the present embodiment is applicable to any apparatus provided with an imaging unit, and may be applied to an information processing apparatus such as a so-called smartphone or tablet terminal.
  • a determination unit that determines whether or not the imaging unit is in a predetermined shielding state;
  • a recognition unit for recognizing a user's operation input according to the predetermined shielding state;
  • An information processing apparatus comprising: (2) The information processing apparatus according to (1), wherein the recognition unit controls whether to recognize the operation input in accordance with a change speed of an image acquired by the imaging unit. (3) The information processing apparatus according to (1), wherein the recognition unit controls whether to recognize the operation input according to a duration of the predetermined shielding state.
  • the said recognition part controls whether the said operation input is recognized according to the measurement result of the distance between the said imaging part and the to-be-photographed object which shields the said imaging part, The said (1).
  • the imaging unit includes a first imaging unit and a second imaging unit,
  • the predetermined shielding state includes a shielding state of the first imaging unit and a shielding state of the second imaging unit,
  • the determination unit performs a first determination regarding a shielding state of the first imaging unit and a second determination regarding a shielding state of the second imaging unit,
  • the recognizing unit recognizes the operation input according to a combination of the first determination and the second determination;
  • the recognizing unit recognizes the operation input according to a timing when the first imaging unit is in the shielding state and a timing when the second imaging unit is in the shielding state, (6)
  • the predetermined shielding state includes a first shielding state and a second shielding state in which the amount of shielding of the angle of view of the imaging unit is smaller than the first shielding state,
  • the recognizing unit is different from an operation input recognized when the imaging unit is determined to be in the first shielding state and is recognized when the imaging unit is determined to be in the second shielding state. Recognize input, The information processing apparatus according to (1).
  • the information processing apparatus according to any one of (1) to (11), further including an output control unit configured to control information related to the operation input to be presented via the output unit.
  • the information processing apparatus wherein the output control unit performs control so that an image captured by the imaging unit is presented via the output unit in accordance with the operation input.
  • the information processing unit according to (12) or (13), wherein the output control unit controls the image according to the position of the imaging unit to be presented via the output unit in accordance with the operation input. apparatus.
  • the output control unit performs control so that a predetermined sound corresponding to the position of the imaging unit is output via the output unit in response to the operation input.
  • the information processing apparatus according to one item.
  • the output control unit controls the notification information that prompts an operation of shielding the imaging unit to be presented via the output unit in accordance with the operation input, any of (12) to (15) The information processing apparatus according to claim 1.
  • the imaging unit is configured to provide an image to a wearable device held on a user's head.
  • the determination unit determines whether substantially the entire angle of view of the imaging unit is shielded according to the brightness of the image captured by the imaging unit. Any one of (1) to (18) The information processing apparatus according to claim 1. (22) The information processing apparatus according to (21), wherein the determination unit determines that substantially the entire angle of view of the imaging unit is shielded when an average luminance value of each pixel of the image is equal to or less than a threshold value. .

Landscapes

  • Engineering & Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

La présente invention a pour objet de reconnaître une entrée d'utilisateur d'une manière plus appropriée sans l'aide d'un dispositif d'entrée disposé dans le boîtier d'un dispositif. À cet effet, l'invention porte sur un dispositif de traitement d'informations comprenant une unité de détermination pour déterminer si une unité d'imagerie est dans un état d'ombrage prescrit, et une unité de reconnaissance pour reconnaître l'entrée d'opération d'un utilisateur conformément à l'état d'ombrage prescrit.
PCT/JP2018/006020 2017-04-27 2018-02-20 Dispositif de traitement d'informations, procédé de traitement d'informations et support d'enregistrement WO2018198499A1 (fr)

Priority Applications (3)

Application Number Priority Date Filing Date Title
JP2019515117A JPWO2018198499A1 (ja) 2017-04-27 2018-02-20 情報処理装置、情報処理方法、及び記録媒体
US16/495,588 US20200042105A1 (en) 2017-04-27 2018-02-20 Information processing apparatus, information processing method, and recording medium
EP18791869.3A EP3617851B1 (fr) 2017-04-27 2018-02-20 Dispositif de traitement d'informations, procédé de traitement d'informations et support d'enregistrement

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2017-088354 2017-04-27
JP2017088354 2017-04-27

Publications (1)

Publication Number Publication Date
WO2018198499A1 true WO2018198499A1 (fr) 2018-11-01

Family

ID=63919781

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2018/006020 WO2018198499A1 (fr) 2017-04-27 2018-02-20 Dispositif de traitement d'informations, procédé de traitement d'informations et support d'enregistrement

Country Status (4)

Country Link
US (1) US20200042105A1 (fr)
EP (1) EP3617851B1 (fr)
JP (1) JPWO2018198499A1 (fr)
WO (1) WO2018198499A1 (fr)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020234939A1 (fr) * 2019-05-17 2020-11-26 ソニー株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations et programme
WO2020241189A1 (fr) * 2019-05-30 2020-12-03 ソニー株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations et programme

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7635008B2 (ja) * 2021-02-01 2025-02-25 キヤノン株式会社 画像処理装置、撮像装置、制御方法およびプログラム
JP2023017615A (ja) * 2021-07-26 2023-02-07 富士フイルムビジネスイノベーション株式会社 情報処理システム及びプログラム
US11983812B2 (en) * 2022-05-31 2024-05-14 Dish Network L.L.C. Marker-based representation of real objects in virtual environments

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010177775A (ja) * 2009-01-27 2010-08-12 Kyocera Corp 携帯電子機器及び音声調整方法
JP2013069040A (ja) * 2011-09-21 2013-04-18 Nippon Telegr & Teleph Corp <Ntt> 命令信号送信装置およびその動作方法
JP2014011504A (ja) * 2012-06-27 2014-01-20 Kyocera Corp 電子機器、制御方法、及び制御プログラム
JP2014186361A (ja) 2013-03-21 2014-10-02 Sony Corp 情報処理装置、操作制御方法及びプログラム

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4381282B2 (ja) * 2004-10-22 2009-12-09 株式会社東芝 携帯端末
US20090174674A1 (en) * 2008-01-09 2009-07-09 Qualcomm Incorporated Apparatus and methods for a touch user interface using an image sensor
JP5685837B2 (ja) * 2010-06-15 2015-03-18 ソニー株式会社 ジェスチャ認識装置、ジェスチャ認識方法およびプログラム
US9335547B2 (en) * 2013-03-25 2016-05-10 Seiko Epson Corporation Head-mounted display device and method of controlling head-mounted display device
KR101616450B1 (ko) * 2014-06-09 2016-05-11 (주) 펀매직 카메라를 통한 가상 버튼 구현방법, 장치 및 컴퓨터 판독가능 기록매체

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2010177775A (ja) * 2009-01-27 2010-08-12 Kyocera Corp 携帯電子機器及び音声調整方法
JP2013069040A (ja) * 2011-09-21 2013-04-18 Nippon Telegr & Teleph Corp <Ntt> 命令信号送信装置およびその動作方法
JP2014011504A (ja) * 2012-06-27 2014-01-20 Kyocera Corp 電子機器、制御方法、及び制御プログラム
JP2014186361A (ja) 2013-03-21 2014-10-02 Sony Corp 情報処理装置、操作制御方法及びプログラム

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP3617851A4

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2020234939A1 (fr) * 2019-05-17 2020-11-26 ソニー株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations et programme
US11846783B2 (en) 2019-05-17 2023-12-19 Sony Group Corporation Information processing apparatus, information processing method, and program
WO2020241189A1 (fr) * 2019-05-30 2020-12-03 ソニー株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations et programme
US11835727B2 (en) 2019-05-30 2023-12-05 Sony Group Corporation Information processing apparatus and information processing method for controlling gesture operations based on postures of user

Also Published As

Publication number Publication date
EP3617851A1 (fr) 2020-03-04
EP3617851B1 (fr) 2021-05-19
US20200042105A1 (en) 2020-02-06
EP3617851A4 (fr) 2020-05-13
JPWO2018198499A1 (ja) 2020-03-05

Similar Documents

Publication Publication Date Title
US11546505B2 (en) Touchless photo capture in response to detected hand gestures
US11520399B2 (en) Interactive augmented reality experiences using positional tracking
JP6747504B2 (ja) 情報処理装置、情報処理方法、及びプログラム
WO2018198499A1 (fr) Dispositif de traitement d&#39;informations, procédé de traitement d&#39;informations et support d&#39;enregistrement
JP7092028B2 (ja) 情報処理装置、情報処理方法、及びプログラム
TW202206978A (zh) 用於擴展現實的隱私控制介面
US11487354B2 (en) Information processing apparatus, information processing method, and program
US12169968B2 (en) Augmented reality eyewear with mood sharing
US11360550B2 (en) IMU for touch detection
US11719931B2 (en) Augmented reality gaming using virtual eyewear beams
CN110895433B (zh) 用于增强现实中用户交互的方法和装置
WO2018146922A1 (fr) Dispositif de traitement d&#39;informations, procédé de traitement d&#39;informations et programme
US20230419439A1 (en) Warping an input image based on depth and offset information
US11263456B2 (en) Virtual object repositioning versus motion of user and perceived or expected delay
CN114115544B (zh) 人机交互方法、三维显示设备及存储介质
CN112578983B (zh) 手指取向触摸检测
JP7616216B2 (ja) 情報処理装置、情報処理方法及びプログラム
US11270409B1 (en) Variable-granularity based image warping
WO2022064881A1 (fr) Dispositif de traitement d&#39;informations, procédé de traitement d&#39;informations, et programme
US11733789B1 (en) Selectively activating a handheld device to control a user interface displayed by a wearable device
CN115705092A (zh) 基于个体的相应位置生成和显示内容
CN117762242A (zh) 在检测到外围设备上的外围事件时对手部手势的抑制

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18791869

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2019515117

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2018791869

Country of ref document: EP

Effective date: 20191127

点击 这是indexloc提供的php浏览器服务，不要输入任何密码和下载