JP6589216B2 - Electronics - Google Patents

Description

  The present invention relates to an electronic device capable of performing predetermined information processing.

  Conventionally, there is a portable game device including a display and a camera provided on the back of the display (for example, Patent Document 1). In such a conventional game device, a captured image is acquired by a user's shutter operation, a characteristic portion of the acquired image is analyzed, and a virtual character is combined with the captured image based on the result and displayed on the display. There is something you can do.

JP 2006-271663 A

  However, the conventional portable device analyzes feature portions based on an image captured by a camera, and is improved in that various inputs are made using the camera and the result of the input is output. There was room for.

  Therefore, one of the objects of the present invention is to provide an electronic device capable of performing various inputs using a camera and outputting a result of the input.

  In the present invention, the following configuration is adopted in order to solve the above-described problems.

  An electronic apparatus according to an embodiment includes a camera, an acquisition unit, a detection unit, and an irradiation unit. The acquisition unit acquires an input image captured by the camera. The detection means detects a gesture made by the user based on the input image acquired by the acquisition means. The irradiating unit irradiates light in the direction of the object related to the gesture based on the detection result by the detecting unit.

  Here, the “gesture made by the user” indicates a gesture / hand gesture performed by the user using a part or all of the face or body, and includes an operating state and a stationary state.

  Based on the above, the electronic device can detect the user's gesture based on the image captured by the camera and irradiate the light based on the detection result.

  In another configuration, the light irradiation direction of the irradiation unit may be the same as the imaging direction of the camera.

  According to the above, light can be irradiated in the same direction as the imaging direction of the camera.

  In another configuration, the detection means may detect a predetermined part of the user's body.

  According to the above, it is possible to detect a predetermined part of the user's body and irradiate light according to the detection result.

  In another configuration, the detection unit may detect a gesture using a user's hand.

  According to the above, it is possible to detect a gesture using the user's hand and irradiate light corresponding to the gesture.

  In another configuration, the electronic device may be a handheld electronic device that a user can hold and operate. The hand-held electronic device is provided with a grip portion that is gripped by one hand of a user, and the detection unit detects a gesture using the other hand of the user.

  According to the above, the user can hold the electronic device with one hand and perform a gesture with the other hand, and the electronic device emits light according to the detection result of the gesture toward the other hand. Can be irradiated.

  In another configuration, the irradiation unit may be a projector.

  Based on the above, it is possible to irradiate light according to the detection result using a projector.

  In another configuration, the irradiating unit may project an image corresponding to a detection result by the detecting unit as irradiating the light.

  Based on the above, it is possible to project an image according to the detection result, for example, it is possible to project an image according to a gesture input.

  In another configuration, the irradiation unit may project the image continuously in accordance with the continuous gesture.

  Based on the above, it is possible to continuously project an image corresponding to a gesture.

  In another configuration, the irradiation unit may project the image onto a predetermined part of the user's body.

  According to the above, when a gesture input is performed using a hand, for example, an image corresponding to the input can be projected onto the input hand.

  In another configuration, the electronic device may further include a display that displays an image. In this case, an image is projected by the irradiation unit and the image is displayed on the display.

  Based on the above, it is possible to project an image by the irradiation means and display the image on the display.

  In another configuration, the electronic apparatus may further include interlocking means for interlocking an image projected by the irradiation means and an image displayed on the display.

  According to the above, the image projected by the projector and the image displayed on the display can be linked.

  In another configuration, the detection unit may detect a position of an object related to the gesture, and the irradiation unit may irradiate light based on the position detected by the detection unit.

  According to the above, light can be irradiated based on the position.

  In another configuration, the electronic device further includes game processing means for performing a predetermined game process based on a detection result by the detection means, and the irradiation means emits light based on the result of the game process. May be.

  Based on the above, it is possible to output an image based on position information of a specific object.

  In another configuration, the electronic device may include an audio output unit that outputs audio in accordance with a detection result by the detection unit.

  According to the above, it is possible to output sound in addition to light output.

  In another configuration, the camera and the irradiation unit may be provided on a side surface of the electronic device.

  According to the above, it is possible to detect the gesture or the movement of the object performed in the side surface direction and irradiate light in the side surface direction based on the detection result.

  In another configuration, the irradiation unit may be a light source that emits light of a predetermined color.

  According to the above, predetermined light can be irradiated using light sources, such as LED, for example.

  An electronic apparatus according to another embodiment includes a camera, an acquisition unit, a detection unit, and an irradiation unit. The acquisition unit acquires an input image captured by the camera. The detection means detects the movement of a specific object based on the input image acquired by the acquisition means. The irradiating unit irradiates light in the direction of the specific object based on the detection result by the detecting unit.

  Here, the “movement of a specific object” may be a movement of a part or all of a human face or body, or may be a movement of an arbitrary object.

  According to an embodiment of the present invention, it is possible to detect a user's gesture or a movement of an object based on an image captured by a camera and irradiate light based on the detection result.

Front view of portable electronic device 1 of the present embodiment Right side view of portable electronic device 1 Rear view of portable electronic device 1 The block diagram which shows an example of an internal structure of the portable electronic device 1 It is a figure which shows an example of the use condition of the portable electronic device 1, and is a figure which shows a mode when a user hold | grips the portable electronic device 1 with both hands The figure which shows a mode that a user performs gesture input using the right hand in the right side surface direction of the portable electronic device 1. The figure which shows a mode that a user performs gesture input using the right hand in the right side surface direction of the portable electronic device 1. The figure which shows an example of the image imaged with the infrared camera 4 at the time of gesture input as shown to FIG. 6A being performed The figure which shows an example of the image imaged with the infrared camera 4 at the time of gesture input as shown to FIG. 6B being performed The figure which shows the recognizable range of the infrared camera 4 The figure which shows the recognizable range of the ranging sensor 5 The figure which shows an example of the process based on the gesture input using the infrared camera 4 The figure which shows an example of the process based on the gesture input using the infrared camera 4 The figure which shows an example of the process based on the gesture input using the infrared camera 4 The figure which shows an example of the process based on the gesture input using the infrared camera 4 The figure which shows an example of the input image imaged with the infrared camera 4 when the gesture input shown to FIG. 10A is performed The figure which shows an example of the input image imaged with the infrared camera 4 when the gesture input shown to FIG. 10B is performed The figure which shows an example of the input image imaged with the infrared camera 4 when the gesture input shown to FIG. 10C is performed The figure which shows an example of the image imaged with the infrared camera 4 when the gesture input shown to FIG. 10D is performed. The figure which shows the other example of the process based on the gesture input using the infrared camera 4. The figure which shows the other example of the process based on the gesture input using the infrared camera 4. The figure which shows an example of the process using the ranging sensor 5 The figure which shows a mode that the image according to a user's gesture input is projected on a user's hand using the projector 8. FIG. The figure which shows a mode that various gesture input is performed with respect to the portable electronic device 1 with the right hand, holding the portable electronic device 1 with the left hand. The flowchart which shows an example of the process performed in the portable electronic device 1 A flowchart showing the predetermined information processing in step S12 of FIG. 16 when an input image from the infrared camera 4 is used. Flowchart showing the predetermined information processing in step S12 of FIG. 16 when processing based on detection information from the distance measuring sensor 5 is performed. Flowchart showing the predetermined information processing in step S12 of FIG. 16 when processing using the infrared camera 4 and the distance measuring sensor 5 is performed. The figure which shows the application example of the process performed in the portable electronic device 1 The figure which shows an example of the image displayed on the display 2 at the time of the process using the outer side camera 9, the infrared camera 4, and / or the ranging sensor 5 being performed. The figure which shows an example of the game which linked the projector 8 and the display 2 The figure which shows an example of the vertically long portable electronic device 50 FIG. 5 is a diagram illustrating an example of a horizontally long portable electronic device The figure which shows a mode when a portable electronic device is rotated 90 degree | times right in the portable electronic device provided with the infrared camera 4, the ranging sensor 5, etc. in the side surface at the time of seeing a screen from the front. The figure which shows an example of the structure by which the infrared camera 4, the ranging sensor 5, etc. were provided in both the side surface and the bottom face A view of a portable electronic device viewed from the bottom when the infrared camera 4 or the distance measuring sensor 5 is tilted The figure which shows a part of image sensor which can image both RGB image and infrared image

  Hereinafter, a portable electronic device according to an embodiment of the present invention will be described. The portable electronic device is a hand-held information processing device that can be operated by a user with his / her hand. For example, the portable electronic device may be a game device, a mobile phone (smartphone or the like), a tablet terminal, a camera, or a wristwatch type terminal. Arbitrary arbitrary apparatuses, such as, may be sufficient. FIG. 1 is a front view of the portable electronic device 1 of the present embodiment. FIG. 2 is a right side view of the portable electronic device 1. FIG. 3 is a rear view of the portable electronic device 1.

  As shown in FIG. 1, the portable electronic device 1 includes a display 2, a touch panel 3, an infrared camera 4, a distance measuring sensor 5, input buttons 6 (6 </ b> A to 6 </ b> D), an irradiation unit 7, and a projector 8. These are housed in the housing 10. The housing 10 (portable electronic device 1) has a plate shape and is a size that can be held by a user with one hand or both hands.

  As the display 2, for example, a liquid crystal display device or an organic EL display device is used, and any other display device may be used. The screen of the display 2 is provided so as to be exposed on the front surface (T5 surface) of the housing 10. The touch panel 3 is provided on the screen of the display 2 and detects a position on the screen touched by the user. As the touch panel 3, one that can detect one point or one that can detect multiple points is used. For example, an arbitrary one such as a capacitance method or a resistance film method may be used.

  The input buttons 6A to 6D accept input (pressing) by the user. The input buttons 6 </ b> A to 6 </ b> D are provided at positions where the user's finger can reach when the user grips both ends of the mobile electronic device 1. Specifically, the input buttons 6A and 6C are in a position where the finger of the right hand can reach when the user holds the portable electronic device 1 with the right hand, the input button 6A is in a position where the thumb of the right hand can reach, and the input button 6C is It is provided at a position where the index finger or middle finger of the right hand can reach. The input buttons 6B and 6D are in a position where the left hand can reach when the user holds the portable electronic device 1 with the left hand, the input button 6B is in a position where the left thumb can reach, and the input button 6D is the left index finger. Alternatively, it is provided at a position where the middle finger can reach. As shown in FIG. 1, the input buttons 6A and 6B are provided on the front surface (T5 surface) of the housing 10, and the input buttons 6C and 6D are provided on the upper surface (T4 surface) of the housing 10. In addition to the input buttons 6A to 6D, an input unit that receives input by the user may be provided with a cross key for inputting a direction, an analog stick, or the like.

  The infrared camera 4 includes a lens and a sensor that is sensitive to light (infrared rays, specifically near infrared rays). The sensor of the infrared camera 4 is an image sensor in which a plurality of elements sensitive to infrared rays are arranged vertically and horizontally, and each element of the image sensor receives infrared rays and converts them into electric signals to output a two-dimensional infrared image. .

  The distance measuring sensor 5 reflects light (for example, infrared light) emitted from a light source provided in the distance measuring sensor 5 to an object, and receives the reflected light with a light receiving element, thereby distance from the object. Measure. As the distance measuring sensor 5, an arbitrary sensor such as a triangular distance measuring method, a TOF (Time Of Flight) method, or the like may be used. As a light source of the distance measuring sensor 5, an LED, a laser diode, or the like that emits infrared light in a specific direction is used.

  The irradiation unit 7 irradiates infrared rays at a predetermined time interval (for example, 1/60 second interval). The irradiation unit 7 emits infrared rays in synchronization with the timing when the infrared camera 4 captures an image. The irradiation unit 7 irradiates infrared rays to a predetermined range in the right side surface direction of the portable electronic device 1. The infrared light emitted by the irradiation unit 7 is reflected by an object, and the reflected infrared light is received by the infrared camera 4, whereby an infrared image is acquired. The irradiation unit 7 may be used for capturing an infrared image by the infrared camera 4 and for measuring a distance by the distance measuring sensor 5. That is, the infrared light from the irradiation unit 7 may be used to capture an image by the infrared camera 4 and the distance sensor 5 may measure the distance.

  The projector 8 has a light source that emits visible light, and projects characters, images, and the like onto a projection surface (a screen, a user's hand as will be described later) using the light from the light source.

  The infrared camera 4, the distance measuring sensor 5, the irradiation unit 7, and the projector 8 are provided on the side surface (for example, the right side surface: T1 surface) of the housing 10. Specifically, the imaging direction (optical axis) of the infrared camera 4 is oriented in a direction perpendicular to the right side surface. The detection direction of the distance measuring sensor 5 and the direction in which the projector 8 emits light are also perpendicular to the right side surface. That is, when the user holds the portable electronic device 1 with the left hand, the infrared camera 4 images the space in the right side surface direction of the portable electronic device 1, and the ranging sensor 5 is in the right side surface direction of the portable electronic device 1. Measure the distance to an object in space. The projector 8 projects an image or the like by irradiating visible light in the same direction as the infrared camera 4 and the distance measuring sensor 5.

  An outer camera 9 is provided on the back surface (T6 surface) of the portable electronic device 1 (FIG. 3). The outer camera 9 can typically capture a direction perpendicular to the imaging direction of the infrared camera 4 and can capture a direction perpendicular to the back surface. The outer camera 9 includes a lens and an image sensor that is sensitive to visible light. The outer camera 9 captures a space in the back direction and acquires a color image (RGB image). In addition to the rear outer camera 9, a camera may be provided on the front, or the rear outer camera 9 is not provided, and a camera is provided on the front (the surface on which the screen of the display 2 is provided). Also good.

  FIG. 4 is a block diagram illustrating an example of the internal configuration of the mobile electronic device 1. As shown in FIG. 4, the portable electronic device 1 includes a vibrator 11, a microphone 12, a speaker 13, a control unit 14, a communication unit 15, a posture detection unit 16, a GPS receiver 17, and a geomagnetic sensor 18 in addition to the above-described units. Prepare. The mobile electronic device 1 includes a battery (not shown), and power is supplied from the battery. These parts are accommodated in the housing 10.

  The control unit 14 includes a display 2, a touch panel 3, an infrared camera 4, a distance measuring sensor 5, an input button 6, an irradiation unit 7, a projector 8, a vibrator 11, a microphone 12, a speaker 13, a communication unit 15, a posture detection unit 16, and a GPS. It is connected to each part such as the receiver 17 and the geomagnetic sensor 18 and controls each part.

  Specifically, the control unit 14 includes a CPU, a memory, and the like, and a predetermined program (for example, a game) stored in a storage device (not shown) (for example, a nonvolatile memory or a hard disk) provided in the portable electronic device 1. Predetermined processing is performed based on processing, image processing, and application programs for performing various calculations. For example, the control unit 14 acquires an image from the infrared camera 4 and analyzes the image, calculates a distance to an object based on a signal from the distance measuring sensor 5, or from the touch panel 3 or the input button 6. The process according to the input signal is performed. Then, the control unit 14 generates an image based on the result of the predetermined process, and outputs the image to the display 2. Note that a program for performing predetermined processing may be downloaded from the outside via the communication unit 15.

  The vibrator 11 operates based on a command from the control unit 14 and vibrates the entire portable electronic device 1. The vibrator 11 is provided at a predetermined position where vibration is easily transmitted to the user's hand (for example, a position biased to either the central portion or the left and right in the housing 10).

  The microphone 12 and the speaker 13 are used for inputting / outputting sound. The communication unit 15 is used for communicating with other devices by a predetermined communication method (for example, wireless LAN or the like). In addition, the posture detection unit 16 is, for example, an acceleration sensor or an angular velocity sensor, and detects the posture of the mobile electronic device 1.

  The GPS receiver 17 receives a signal from a GPS (Global Positioning System) satellite, and the portable electronic device 1 can calculate its own position by receiving the signal. For example, when a predetermined operation (for example, gesture input or button input using an infrared camera 4 to be described later, operation of shaking the mobile electronic device 1) is performed at a specific position, the mobile electronic device 1 An object associated with a specific position may be displayed. For example, when a game is played in the mobile electronic device 1, when the mobile electronic device 1 is at a specific position, an object corresponding to the specific position may appear in the game.

  The geomagnetic sensor 18 is a sensor capable of detecting the direction and magnitude of magnetism. For example, the mobile electronic device 1 determines whether or not it is facing a specific direction based on the detection result of the geomagnetic sensor 18, and when a predetermined operation (such as the above gesture input) is performed in the specific direction. The object may be displayed. For example, when a game is played on the mobile electronic device 1, an object corresponding to a specific direction may appear in the game. Moreover, the portable electronic device 1 may combine GPS information using the GPS receiver 17 and azimuth information using a geomagnetic sensor. For example, when the mobile electronic device 1 is at a specific position and is facing a specific direction, an object corresponding to the specific position and the specific direction is displayed, or the object is made to appear in the game. Also good.

  FIG. 5 is a diagram illustrating an example of a usage state of the mobile electronic device 1 and is a diagram illustrating a state when the user holds the mobile electronic device 1 with both hands. As shown in FIG. 5, the user grips both left and right ends of the mobile electronic device 1. When the user grips both left and right ends, the user can press the input button 6B with the thumb of the left hand and can press the input button 6A with the thumb of the right hand. Further, when the user grips both left and right ends, the user can move the index finger or middle finger of the left hand, and can press the input button 6D with the index finger or middle finger of the left hand, and moves the index finger or middle finger of the right hand The input button 6C can be pressed with the index finger or middle finger of the right hand.

  Next, input to the portable electronic device 1 will be described. In the present embodiment, the user can perform gesture input on the mobile electronic device 1 using the right hand while holding the mobile electronic device 1 with the left hand.

  6A and 6B are diagrams illustrating a state where the user performs gesture input using the right hand in the right side surface direction of the mobile electronic device 1. FIG. 7A is a diagram illustrating an example of an image captured by the infrared camera 4 when the gesture input illustrated in FIG. 6A is performed. FIG. 7B is a diagram illustrating an example of an image captured by the infrared camera 4 when the gesture input illustrated in FIG. 6B is performed.

  As illustrated in FIGS. 6A and 6B, the user performs various gesture inputs using the right hand at a position in the right side surface direction of the mobile electronic device 1. The portable electronic device 1 captures an infrared image with an infrared camera 4 provided on the right side surface of the portable electronic device 1 and analyzes the captured infrared image to determine a user's gesture input.

  Specifically, when the image shown in FIG. 7A or 7B is acquired from the infrared camera 4, the portable electronic device 1 displays a specific object (specifically, a human hand) included in the acquired image. The type of gesture performed by the user is determined based on the detected shape and the like of the specific object. For example, the mobile electronic device 1 determines whether or not an object having a predetermined shape exists in the acquired image by pattern matching or the like. For example, when the image shown in FIG. 7A is acquired, the mobile electronic device 1 can recognize “goo” from the number of fingers, and when the image shown in FIG. "Par" can be recognized. And the portable electronic device 1 performs the process according to the kind of gesture so that it may mention later.

  Here, the determined gesture is a gesture / hand gesture using a part or the whole of the user's hand or face, and the portable electronic device 1 recognizes the stationary state of the hand as a gesture input. Alternatively, a series of actions using the hand may be recognized as a gesture input. In addition, the mobile electronic device 1 may recognize a gesture input performed while the user holds an object. In this case, the mobile electronic device 1 may recognize a stationary state or an operating state of only an object held by the user as a gesture input, or a gesture of an entire stationary state or an operating state including the user's hand and the object. It may be recognized as an input.

  Moreover, the portable electronic device 1 of this embodiment calculates the distance between the portable electronic device 1 and an object using the ranging sensor 5 so that it may mention later, and performs a process based on the calculated distance. For example, the mobile electronic device 1 detects the presence or absence of an object in the right side surface direction of the mobile electronic device 1 based on information from the distance measuring sensor 5, or detects the distance to the object to detect the right side surface. The movement of the object in the direction can be detected. For example, when the distance detected by the distance measuring sensor 5 has changed within a predetermined time, the portable electronic device 1 recognizes that the user is shaking his / her hand in the right / left direction of the portable electronic device 1. be able to. If the object is detected or not detected by the distance measuring sensor 5 within a predetermined time, the portable electronic device 1 can recognize that the user is waving his hand up and down. Then, the portable electronic device 1 performs a predetermined process according to the detected movement of the object, and displays the result of the process on the display 2.

  As described above, in the present embodiment, by using the infrared camera 4 and / or the distance measuring sensor 5 to detect a specific object in the right side surface direction of the mobile electronic device 1, various gesture inputs to the mobile electronic device 1 are performed. And the result of processing corresponding to the input can be displayed on the display 2.

  Here, each of the infrared camera 4 and the distance measuring sensor 5 has a recognizable range in which an object can be recognized. FIG. 8 is a diagram showing a recognizable range of the infrared camera 4. FIG. 9 is a diagram showing a recognizable range of the distance measuring sensor 5.

  As shown in FIG. 8, the infrared camera 4 has a predetermined angle of view θ. The infrared camera 4 images a range (for example, a range of several tens to 180 degrees) included in the angle of view θ of the infrared camera 4 in the right side surface direction of the portable electronic device 1, and the range uses the infrared camera 4. This is the recognizable range of gesture input. Here, when the distance between the object to be recognized and the portable electronic device 1 is too short, the entire object cannot be imaged using the infrared camera 4. Therefore, an area away from the mobile electronic device 1 by a predetermined distance d in the range included in the angle of view θ is a recognizable range of the infrared camera 4. The distance d varies depending on the size of the recognition target object.

  Further, as shown in FIG. 9, the distance measuring sensor 5 measures whether or not an object exists in the right side surface direction of the portable electronic device 1 and the distance to the object. For example, the distance measuring sensor 5 can detect a distance of about several cm to 1 m. Unlike the infrared camera 4, the recognizable range of the distance measuring sensor 5 extends substantially linearly from the vicinity of the portable electronic device 1 toward the right side surface. That is, the distance measuring sensor 5 detects an object that exists at a position substantially on a straight line in the detection direction, and has extremely high directivity, while the infrared camera 4 captures an object that exists within a certain range. The directivity is low.

  Next, processing based on gesture input detected using the infrared camera 4 and the distance measuring sensor 5 will be described.

  10A to 10D are diagrams illustrating an example of processing based on gesture input using the infrared camera 4. As shown in FIG. 10A, when the user holds the mobile electronic device 1 with the left hand and makes the right hand “goo” at a predetermined position in the right side surface direction of the mobile electronic device 1, the character 21 is displayed on the display 2. As shown in FIG. 10B from the state of FIG. 10A, when the user puts the right hand close to the right side of the portable electronic device 1 with the right hand set to “goo”, the character 21 of the display 2 is enlarged.

  Furthermore, from the state shown in FIG. 10B, when the user changes the right hand to “par”, the expression of the character 21 on the display 2 changes (FIG. 10C). For example, as shown in FIG. 10C, the character 21 changes to a smiley expression. Furthermore, as shown in FIG. 10D, when the user puts the right hand close to the portable electronic device 1 with the right hand set to “par”, the character 21 is further enlarged and the facial expression of the character 21 changes. At this time, the vibrator 11 is activated to vibrate the portable electronic device 1, and sound may be output from the speaker 13.

  FIG. 11A is a diagram illustrating an example of an input image captured by the infrared camera 4 when the gesture input illustrated in FIG. 10A is performed. FIG. 11B is a diagram illustrating an example of an input image captured by the infrared camera 4 when the gesture input illustrated in FIG. 10B is performed. FIG. 11C is a diagram illustrating an example of an input image captured by the infrared camera 4 when the gesture input illustrated in FIG. 10C is performed. FIG. 11D is a diagram illustrating an example of an image captured by the infrared camera 4 when the gesture input illustrated in FIG. 10D is performed.

  When an input image as shown in FIG. 11A is acquired from the infrared camera 4, the portable electronic device 1 detects the object 30 included in the acquired input image, and based on the shape of the detected object 30, Determine the type of gesture. The portable electronic device 1 stores a gesture type and a character 21 displayed on the display 2 in association with each other, and the portable electronic device 1 displays an image of the character 21 corresponding to the determined gesture type. Is displayed on the display 2 (FIG. 10A). Note that the vibrator 11 may be operated to vibrate the portable electronic device 1 according to the determined type of gesture.

  Next, when the input image as illustrated in FIG. 11B is acquired, the mobile electronic device 1 detects the object 30 included in the acquired input image, and the type of gesture based on the shape of the detected object 30. Is determined. In FIG. 11B, since it is discriminated as the same gesture as FIG. 11A, the image of the character 21 displayed on the display 2 is the same image as FIG. 10A (FIG. 10B). However, the closer the object (user's right hand) is to the portable electronic device 1, the larger the character 21 displayed on the display 2 is. Specifically, the mobile electronic device 1 displays on the display 2 by comparing the size of the object 30 included in the input image illustrated in FIG. 11A with the size of the object 30 included in the input image illustrated in FIG. 11B. The size of the image of the character 21 to be determined is determined. As described above, the mobile electronic device 1 is based on the size of the specific object included in the image acquired from the infrared camera 4, and the relative distance (relative position) between the mobile electronic device 1 and the specific object. And the size of the character 21 displayed on the display 2 is determined according to the relative distance. At this time, you may vibrate the portable electronic device 1 according to relative distance.

  Next, when an input image as illustrated in FIG. 11C is acquired from the infrared camera 4, the mobile electronic device 1 detects the object 30 included in the input image, and performs the gesture based on the shape of the detected object 30. Determine the type. Based on the shape of the object 30 included in the input image illustrated in FIG. 11C, the mobile electronic device 1 determines that the gesture is “par”, and as illustrated in FIG. 10C, the character 21 is different from that illustrated in FIGS. 10A and 10B. The image is displayed on the display 2. The size of the character 21 displayed at this time may be determined to be the same size as the character 21 determined immediately before or may be determined based on the size of the object included in the input image. .

  Furthermore, when an input image as illustrated in FIG. 11D is acquired from the infrared camera 4, the mobile electronic device 1 detects the object 30 included in the input image, and the type of gesture based on the detected shape of the object 30. Is determined. However, in FIG. 11D, a part of the object 30 is missing, and the mobile electronic device 1 cannot accurately determine the type of gesture. For this reason, as shown in FIG. 10D, the mobile electronic device 1 displays the character 21 in a mode indicating that the type of gesture cannot be determined on the display 2 and uses the mobile electronic device 1 to notify the user of a detection error. Vibrate or output a warning sound.

  As shown in FIGS. 10A to 10D, the user performs a gesture input on the mobile electronic device 1 by moving the right hand. However, even when the mobile electronic device 1 itself is moved with the right hand fixed, there is no gesture input. May be determined to have occurred. In order to avoid this, for example, when the movement of the portable electronic device 1 itself is detected using an acceleration sensor or an angular velocity sensor, and the movement of the portable electronic device 1 itself is equal to or greater than a predetermined threshold, no gesture input is performed. It may be determined that For example, when it is determined whether or not the mobile electronic device 1 itself is moving using a detection value from an acceleration sensor or the like, and it is determined that the mobile electronic device 1 itself is moving in the right direction shown in FIG. 10A It may be determined that the gesture input for moving the right hand in the left direction has not been performed. Further, for example, when movement of the right hand is detected based on an image acquired from the infrared camera 4, if movement of the mobile electronic device 1 itself is detected using an acceleration sensor or the like, the mobile electronic device 1 itself Depending on the movement, a process of canceling the movement of the right hand detected based on the image may be performed, and the character 21 may be displayed based on the movement of the right hand calculated by the process. That is, predetermined information processing may be performed based on the movement of the right hand based on the image acquired from the infrared camera 4 and the movement of the mobile electronic device 1 itself detected using an acceleration sensor or an angular velocity sensor. . Further, whether or not the mobile electronic device 1 itself is moving may be determined based on an image from the outer camera 9, and if the mobile electronic device 1 itself is moving, it may be determined that no gesture input has been performed. .

  12A and 12B are diagrams illustrating another example of processing based on gesture input using the infrared camera 4. In this example, a calculation problem is displayed on the display 2. The user answers the problem displayed on the display 2 using the right hand. For example, as shown in FIG. 12A, when the character string “1 + 4 =?” Is displayed, when the user performs a gesture input (“par” gesture) indicating the answer “5” of the question with the right hand, The display 2 shows that the problem has been answered correctly. At this time, a sound indicating that the answer is correct may be output, or the vibrator 11 may be operated with a pattern indicating that the answer is correct to vibrate the portable electronic device 1.

  Also, as shown in FIG. 12B, when the character string “1 + 1 =?” Is displayed, the user inputs a gesture input with the right hand that is different from the answer “2” (input “5” with the gesture). If it does, the display 2 will show that it is an incorrect answer. At this time, a voice indicating that the answer is incorrect may be output, or the vibrator 11 may be operated with a pattern indicating that the answer is incorrect.

  As described above, the mobile electronic device 1 determines the gesture input performed by the user based on the input image from the infrared camera 4 and determines whether or not the problem is correctly answered. Then, the mobile electronic device 1 displays the determination result on the display 2. For example, the mobile electronic device 1 displays a plurality of problems in order within a predetermined time, and the user answers the displayed problems with a gesture using the right hand. Then, for example, the score is displayed on the display 2 based on how many questions are answered correctly within a predetermined time.

  Next, an example of processing using the distance measuring sensor 5 will be described. FIG. 13 is a diagram illustrating an example of processing using the distance measuring sensor 5. As shown in FIG. 13, when the user holds the portable electronic device 1 with the left hand and holds the right hand at a predetermined position in the right side surface direction of the portable electronic device 1, the snake character 22 appears from the right end of the display 2. When the user brings his right hand closer to the portable electronic device 1, the snake character 22 extends from the right end of the display 2 to the left. Specifically, the mobile electronic device 1 calculates the distance between the mobile electronic device 1 and the user's right hand based on the information from the distance measuring sensor 5, and the length of the snake character 22 is determined according to the calculated distance. To decide. Then, the mobile electronic device 1 displays the snake character 22 on the display 2. Moreover, the portable electronic device 1 may vibrate the portable electronic device 1 by operating the vibrator 11 according to the distance.

  As described above, the portable electronic device 1 calculates the distance to the object existing in the right side surface direction of the portable electronic device 1 using the distance measuring sensor 5, and displays an image corresponding to the calculated distance on the display 2. indicate. In the processing using the distance measuring sensor 5, predetermined processing is performed based on whether or not an object exists and / or the distance to the object. Further, as described above, the movement of the object in the right side surface direction of the portable electronic device 1 is detected based on the detection of the object and the change in the distance at a predetermined time, and a predetermined process is performed according to the detection of the movement. .

  Next, an example of output using the projector 8 will be described. FIG. 14 is a diagram illustrating a state in which an image corresponding to a user's gesture input is projected onto the user's hand using the projector 8. As shown in the upper diagram of FIG. 14, at the predetermined position in the right side surface direction of the mobile electronic device 1, for example, when the user changes his / her right hand to “par”, the character 21 is projected onto the user's right hand. Specifically, the mobile electronic device 1 determines the type of gesture input performed by the user by analyzing an input image captured by the infrared camera 4. Next, the mobile electronic device 1 generates an image of the character 21 based on the determined gesture input, and outputs the image of the character 21 using the projector 8. At this time, the same (or different) image may be output to the display 2.

  Further, as shown in the lower diagram of FIG. 14, when the right hand is moved away from the portable electronic device 1, for example, the image of the character 21 is displayed by the user using the projector 8 so that the size of the character 21 projected on the right hand does not change. Projected to the right hand. Specifically, the mobile electronic device 1 analyzes the image captured by the infrared camera 4 to determine the user's gesture input, and generates an image according to the determination result. Further, the portable electronic device 1 determines the size of the image to be output based on the size of the hand included in the input image from the infrared camera 4. The mobile electronic device 1 may determine the size of the image to be output based on the distance measured by the distance measuring sensor 5 instead of or in addition to the input image from the infrared camera 4. Then, the portable electronic device 1 projects an image of the determined size using the projector 8. At this time, the same (or different) image may be output to the display 2. When the user performs a gesture (for example, “goo”) different from that in FIG. 14 using the right hand, an image different from that in FIG. 14 is projected on the right hand. That is, different images are projected according to the type of gesture.

  For example, the mobile electronic device 1 sequentially displays gestures to be performed by the user on the display 2, and when the user performs the same gesture as the displayed gesture within a predetermined time, the mobile electronic device 1 determines success and what is the displayed gesture? If the user makes a different gesture, a failure determination is performed. And the portable electronic device 1 projects the image according to a determination result on a user's hand by outputting an image from the projector 8 according to a determination result.

  Note that the user's hand may be tracked by imaging the user's hand with the infrared camera 4 and the projector 8 may be controlled so that the image is projected onto the user's hand. The projector 8 can project an image within a predetermined range. The portable electronic device 1 acquires an image from the infrared camera 4 at predetermined time intervals, recognizes the position of the user's hand based on the image, and determines a position to project the image. For example, when the hand is moved upward in a state where the image is projected on the user's hand as shown in the lower diagram of FIG. 14, the mobile electronic device 1 moves the hand upward based on the image from the infrared camera 4. Detect that you have moved to. Based on the detection result, the portable electronic device 1 projects an image of the object on the user's hand after movement using the projector 8. In this way, the mobile electronic device 1 acquires images from the infrared camera 4 at predetermined time intervals, so that even when the user's hand is moving, the image is displayed so as to follow the movement of the user's hand. Can be projected onto the hand. Moreover, the portable electronic device 1 can project an image continuously according to a user's continuous gesture.

  Alternatively, the user may be notified of the range in which the user can input a gesture by irradiating light in the right side surface direction of the portable electronic device 1 using the projector 8. For example, when the entire hand of the user can be imaged by the infrared camera 4, blue light may be emitted, and when only a part of the user's hand can be imaged, red light may be emitted. That is, by irradiating light from the projector 8 to a range that is the same as or similar to the imaging range (view angle) of the infrared camera 4, the user can recognize the range in which gesture input is possible, and can guide the user. . Instead of the projector 8 capable of projecting an image, a light source that emits predetermined light (for example, an LED, a halogen lamp, a fluorescent lamp, an EL lamp, or the like) is provided in the portable electronic device 1 and emits light from the light source. Thus, the user may be notified of the range in which gesture input is possible.

  As described above, in the present embodiment, not only input using the infrared camera 4 and the distance measuring sensor 5 but also light output using the projector 8 can be performed from the side surface direction of the portable electronic device 1.

  As described above, in the present embodiment, the infrared camera 4 and the distance measuring sensor 5 are provided on the right side surface of the portable electronic device 1, and the user performs various gesture inputs with the right hand while holding the portable electronic device 1 with the left hand. A predetermined process corresponding to the gesture input is performed. Since the infrared camera 4 and the distance measuring sensor 5 are provided on the right side, the user can perform gesture input with the other hand without a sense of incongruity while grasping the portable electronic device 1 with one hand and viewing the display 2. it can.

  For example, when the infrared camera 4 and the distance measuring sensor 5 for gesture input are provided on the T5 surface (front) on which the screen of the display 2 is provided, when the user performs the gesture input, Overlap will reduce the visibility of the display 2. In addition, when the infrared camera 4 and the distance measuring sensor 5 for gesture input are provided on the T6 surface (rear surface) opposite to the screen of the display 2, the user performs gesture input on the rear surface of the screen. The right hand is not visible, making it difficult to input gestures.

  However, in the portable electronic device 1 of the present embodiment, an infrared camera 4 and a distance measuring sensor 5 for inputting a gesture are provided on the side surface of the display 2. For this reason, even when the user performs gesture input with the right hand, the screen and the right hand do not overlap, and the gesture input can be performed without hindering the visibility of the screen. In addition, since an infrared camera or the like is provided on the side surface of the display 2, it is possible to perform gesture input to the portable electronic device 1 with one hand while holding the portable electronic device 1 with one hand, and from both the left and right directions. It is possible to input to the portable electronic device 1 so as to be sandwiched, or to confirm an output (image display) corresponding to the input. As a result, it is possible to give the user a new feeling of operation and feeling of use with respect to the portable electronic device.

  FIG. 15 is a diagram illustrating how various gestures are input to the mobile electronic device 1 with the right hand while holding the mobile electronic device 1 with the left hand. As shown in FIG. 15, the user can move the right hand to the left and right, move it away, move it up and down, rotate it, and extend or bend any of the five fingers of the right hand. It is possible to input from the side direction to the portable electronic device 1. By performing the input from the side direction, the user can feel more strongly that the input is performed on the mobile electronic device 1.

  The user can perform a gesture input from the right hand direction while viewing the screen of the display 2 and can visually recognize an image displayed on the display 2 as a result of the gesture input. That is, at the same time as the user sees the screen of the display 2, the right hand that performs gesture input also enters the field of view, and the user can visually recognize the input using the right hand and the output from the display 2 at the same time. You can go and check the output accordingly.

  In addition, the portable electronic device 1 according to the present embodiment is robust because the infrared camera 4 is used instead of a normal camera that captures a visible light image (RGB image). That is, when a normal camera is used, depending on the brightness of the external environment (intensity of visible light), the amount of light may be too much or too little to obtain an input image suitable for image recognition. . In addition, when recognizing a human hand when using a normal camera, the hand is recognized by the shape and skin color of the hand, but the human hand cannot be recognized by the light color of the external environment. Sometimes. That is, depending on the external environment, an image capable of detecting a specific object (for example, a user's hand) may not be acquired. However, when the infrared camera 4 is used, by passing infrared light from the irradiation unit 7 (infrared light irradiated from the irradiation unit 7 and reflected by the object) while blocking visible light with a filter, An input image suitable for image recognition can be acquired without being affected by the external environment. In particular, the amount of infrared light is reduced indoors where sunlight is blocked, and by irradiating a specific object with infrared light from the irradiating unit 7, the specific object is not affected by the external environment. A clear image including the image can be acquired. Even outdoors, for example, at night, an image necessary for image recognition can be acquired by irradiating a specific object with infrared light from the irradiation unit 7.

(Explanation of flowchart)
Next, an example of processing performed in the mobile electronic device 1 will be described. FIG. 16 is a flowchart illustrating an example of processing performed in the mobile electronic device 1. The process shown in FIG. 16 is performed by the control unit 14 executing a predetermined program. Further, the process shown in FIG. 16 is repeatedly performed at a predetermined time interval (for example, once every 1/60 seconds).

  As shown in FIG. 16, in step S <b> 10, the control unit 14 acquires an input image from the infrared camera 4. Specifically, the control unit 14 causes the irradiation unit 7 to irradiate infrared rays and acquires an input image captured by the infrared camera 4. Next, the control unit 14 acquires detection information from the distance measuring sensor 5 (step S11).

  Next, the control unit 14 performs predetermined information processing (step S12). Here, as predetermined information processing, processing using an input image from the infrared camera 4, processing based on detection information from the distance measuring sensor 5, and both the infrared camera 4 and the distance measuring sensor 5 are used. Any of the processes can be considered. The predetermined information processing in step S12 will be described with reference to FIGS.

  After the processing in step S12, the control unit 14 outputs an image based on the result of the predetermined information processing in step S12 to the display 2 and / or the projector 8 (step S13). And the control part 14 determines whether the process shown in FIG. 16 is complete | finished (step S14), when it determines with not complete | finishing, when the process of step S10 is performed again and it determines with complete | finishing, The process shown in FIG. 16 ends.

  Next, the predetermined information processing in step S12 of FIG. 16 will be described. As the predetermined information processing, as described above, processing using an image from the infrared camera 4, processing based on detection information from the distance measuring sensor 5, and both the infrared camera 4 and the distance measuring sensor 5 are used. There is processing that was. Which of these three processes is performed depends on the type of application. Below, each process is demonstrated with reference to FIGS.

(Predetermined information processing 1 using the infrared camera 4)
FIG. 17 is a flowchart showing the predetermined information processing in step S12 of FIG. 16 when an input image from the infrared camera 4 is used. The process illustrated in FIG. 17 is a process for executing the application illustrated in FIGS. 10 and 12, for example.

  As shown in FIG. 17, the control unit 14 performs image recognition processing on the input image from the infrared camera 4 acquired in step S10 (step S20). Specifically, the control unit 14 detects the user's hand (specific object) included in the input image from the infrared camera 4 using a predetermined image recognition technique.

  Next, the control unit 14 determines whether or not the user's hand has been detected in the image recognition process (step S21). When the user's hand is detected (step S21: YES), the control unit 14 determines the type of gesture (step S22). Specifically, based on the detected shape of the user's hand, the control unit 14 determines the type of gesture corresponding to the shape. Subsequently, the control unit 14 performs a process according to the determination result of the gesture (step S23). For example, the control unit 14 selects an image corresponding to the determined gesture type, or determines success or failure in a predetermined game (determines whether or not the input is correct) according to the determined gesture type.

  The content of the process in step S23 differs depending on the type of program (application) to be executed. For example, an arbitrary game program may be executed as the program. In addition, a program for performing an operation on the mobile electronic device 1 (for example, enlargement, reduction, scrolling, etc., activation or termination of a predetermined application, setting of the mobile electronic device 1, etc.) may be executed. For example, when a program corresponding to the process illustrated in FIG. 10 is executed, the control unit 14 determines the type of image to be output based on the determined type of gesture and the size of the user's hand included in the input image. As well as the size of the image. For example, when a program corresponding to the process shown in FIG. 12 is executed, the control unit 14 determines whether or not the answer made by the user is correct (whether or not a correct gesture has been made).

  Next, the control part 14 produces | generates an output image according to the result of the process of step S23 (step S24). The output image generated here is output to the display 2 and / or the projector 8 in step S13.

  On the other hand, when it determines with a specific object not being detected in step S21 (step S21: NO), the control part 14 complete | finishes the process shown in FIG.

(Predetermined information processing 2 using the distance measuring sensor 5)
FIG. 18 is a flowchart showing the predetermined information processing in step S12 of FIG. 16 when processing based on detection information from the distance measuring sensor 5 is performed. The process illustrated in FIG. 18 is a process for executing the application illustrated in FIG. 13, for example.

  As shown in FIG. 18, the control unit 14 calculates the distance between the object and the portable electronic device 1 based on the detection information from the distance measuring sensor 5 acquired in step S11 (step S30). Next, the control unit 14 determines whether the distance has been calculated (step S31). When the distance cannot be calculated (step S31: NO), that is, when an object is not detected, the control unit 14 ends the process illustrated in FIG.

  When the distance can be calculated (step S31: YES), the control unit 14 performs a process according to the distance (step S32).

  The content of the process in step S32 varies depending on the type of program (application) to be executed. For example, an arbitrary game program may be performed as the program. In addition, a program for performing an operation on the mobile electronic device 1 may be executed. For example, when a program corresponding to the process shown in FIG. 13 is executed, the control unit 14 determines the length of the snake object 22 based on the calculated distance. In step S32, the movement of the object may be detected based on the distance calculated in the past predetermined period. That is, when an object has not been detected or the calculated distance has changed in a predetermined past period, the control unit 14 detects the movement of the object based on the change pattern. The process according to the detected movement is performed.

  And the control part 14 produces | generates an output image according to the result of the process of step S32 (step S33). The output image generated here is output to the display 2 and / or the projector 8 in step S13.

(Predetermined information processing 3 using the infrared camera 4 and the distance measuring sensor 5)
Next, predetermined information processing using both the infrared camera 4 and the distance measuring sensor 5 will be described. FIG. 19 is a flowchart showing the predetermined information processing in step S12 of FIG. 16 when processing using the infrared camera 4 and the distance measuring sensor 5 is performed. 19, the same processes as those in FIGS. 17 and 18 are denoted by the same reference numerals, and the description thereof is omitted.

  As shown in FIG. 19, following the image recognition process (step S20), when the user's hand (specific object) is detected in the input image from the infrared camera 4 (step S21: YES), the control unit 14 Determines the type of gesture (step S22). Subsequently, the control unit 14 determines whether or not the type of gesture has been normally determined (step S40). For example, as illustrated in FIG. 11D, even when a specific object is detected in an image included in the image from the infrared camera 4, if a part of the specific object is missing, the control unit 14 Determines “NO” in step S40.

  When the gesture can be normally determined (step S40: YES), the control unit 14 performs processing according to the determination result (step S23).

  On the other hand, when the gesture cannot be normally determined (step S40: NO), or when the user's hand is not detected (step S21: NO), the control unit 14 uses the detection information from the distance measuring sensor 5 as the detection information. Based on this, a distance is calculated (step S30). Thereafter, the control unit 14 determines whether or not the distance has been calculated as in FIG. (Step S31). If the distance can be calculated (step S31: YES), the control unit 14 performs processing according to the distance (step S32).

  When the process of step S23 is executed, or when the process of step S32 is executed, the control unit 14 generates an output image according to the process of step S23 or the result of the process of step S32 (step S41). That is, when the process of step S23 is executed, an output image corresponding to the process of step S23 is generated. When the process of step S32 is executed, an output image corresponding to the process of step S32 is generated. Is done. The generated output image is output to the display 2 and / or the projector 8 in step S13.

  As described above, in the process shown in FIG. 19, the process is performed using both the infrared camera 4 and the distance measuring sensor 5. In the process shown in FIG. 19, the infrared camera 4 and the distance measuring sensor 5 are complementary to each other when the same process is performed. For example, when the distance between the mobile electronic device 1 and the specific object is too short and the infrared camera 4 cannot recognize the gesture input normally, the process is performed based on the detection result of the distance measuring sensor 5. That is, when a specific object exists in a range that cannot be recognized by the infrared camera 4, the specific object is detected using the distance measuring sensor 5. Conversely, when a specific object exists in a range that cannot be recognized by the distance measuring sensor 5, the specific object may be detected using the infrared camera 4.

  Thus, one of the infrared camera 4 and the distance measuring sensor 5 is selected according to the detection situation. Specifically, when one detection result is not suitable for the predetermined process, the infrared camera 4 and the distance measuring sensor 5 can complement each other by performing the same process using the other detection result. Thereby, the gesture input performed by the user can be more reliably discriminated, and the operability can be improved.

  In the case where both the infrared camera 4 and the distance measuring sensor 5 are used, a predetermined process may be performed based on both detection results. That is, both the infrared camera 4 and the distance measuring sensor 5 may be selected and used for predetermined processing. For example, when a specific object exists in the recognizable range of the infrared camera 4, based on the image of the specific object detected by the infrared camera 4 and the distance to the specific object calculated by the distance measuring sensor 5. A predetermined process may be performed.

  Moreover, the process shown with the flowchart of the said embodiment is a mere example, and a part of said process may not be performed and the other process of the said process may be added. Further, any order of processing may be used.

  As described above, in the portable electronic device 1 of the present embodiment, when the display 2 is viewed from the front, the infrared camera 4 and the distance measuring sensor 5 are provided on the side surface (right side surface). The sensor 5 can be used to perform gesture input using the user's hand, and an output image corresponding to the gesture input can be output to the display 2 or the projector 8. When the infrared camera 4 is used, various gestures can be identified at a position away from the portable electronic device 1 by a predetermined distance, and when the distance measuring sensor 5 is used, the object can be detected in the vicinity of the portable electronic device 1. Presence / absence can be determined, or predetermined processing can be performed based on the distance between the object and the portable electronic device 1. Moreover, a predetermined process can be performed using both the infrared camera 4 and the distance measuring sensor 5. Then, an image as a result of the predetermined process can be displayed on the display 2, the image can be projected using the projector 8, and the portable electronic device 1 can be vibrated according to the result of the predetermined process. .

  In this embodiment, since the infrared camera 4 and the distance measuring sensor 5 are used, it is possible to recognize various gesture inputs at low cost while suppressing power consumption. That is, the irradiation unit 7 instantaneously irradiates infrared light at predetermined time intervals in synchronization with the image capturing timing of the infrared camera 4. For this reason, a clear image can be picked up even if the light irradiation time is short, so that power consumption can be suppressed. Further, when the distance measuring sensor 5 is used, it is not necessary to irradiate light in a wide range, and the absolute distance to the object to be detected can be calculated only by irradiating light in a limited specific direction. , Power consumption can be suppressed. For example, when a distance image sensor capable of measuring a distance for each pixel together with a two-dimensional image is used, the irradiation unit needs to irradiate light over a wide range and calculate the distance for each pixel. Increases load and power consumption. The distance image sensor is more expensive than the distance measuring sensor. However, when the infrared camera 4 and the distance measuring sensor 5 are used, cost and power consumption can be suppressed, while various gesture inputs can be discriminated.

(Application examples)
In addition to the processing described above, the following processing may be performed using the mobile electronic device 1.

  FIG. 20 is a diagram illustrating an application example of processing performed in the mobile electronic device 1. For example, as shown in FIG. 20, when a gesture input for moving the right hand toward the mobile electronic device 1 is performed, a ball appears from the right end of the screen of the display 2 in accordance with the gesture input and moves to the left. . At this time, the faster the right hand moves, the faster the ball moves. Specifically, the mobile electronic device 1 calculates the absolute distance to the object using the distance measuring sensor 5, calculates the speed of the object based on the change in the distance, and moves the ball based on the calculated speed. To decide. For example, the mobile electronic device 1 may calculate the relative distance to the object based on the image from the infrared camera 4, and may calculate the speed of the object based on the change in the relative distance. When the user presses the input button 6B with the left hand (or touches the screen of the display 2) at the timing when the ball reaches the left end of the screen, the ball rebounds in the opposite direction, and the timing at which the input button 6B is pressed is shifted. The ball will not bounce in the opposite direction. When the user presses the input button 6B with the left hand at the timing when the ball reaches the left end of the screen, the ball rebounds in the opposite direction, and the vibrator 11 may be operated to vibrate the left hand or output sound. Also good. As described above, an application may be executed in which an image is displayed on the display 2 in response to a gesture input performed with the right hand, and input is performed using the left hand after a predetermined time has elapsed.

  In another application example, for example, a predetermined image may be displayed using an input using the infrared camera 4 and / or the distance measuring sensor 5 and the outer camera 9. Specifically, the user's gesture input is detected using the infrared camera 4 and / or the distance measuring sensor 5 while displaying the image captured by the outer camera 9 on the display 2 in real time, and the gesture input is made according to the gesture input. The image may be displayed superimposed on the image captured by the outer camera 9.

  FIG. 21 is a diagram illustrating an example of an image displayed on the display 2 when processing using the outer camera 9, the infrared camera 4, and / or the distance measuring sensor 5 is performed.

  As shown in FIG. 21, for example, when a predetermined marker 60 is placed in the real space and the marker 60 is imaged by the outer camera 9, an image captured by the outer camera 9 (actual camera) is displayed on the display 2. At the same time, a virtual object is superimposed and displayed at the position of the marker included in the captured image. The user operates the virtual object by a gesture input with the right hand. For example, when the right hand is moved in the direction of the arrow in FIG. 21, the virtual object may be moved in the left direction. Specifically, the portable electronic device detects the position and orientation of the marker from the captured image acquired from the real camera, and calculates the positional relationship between the real camera and the marker. Based on the calculated positional relationship, the mobile electronic device sets a virtual space coordinate system (for example, a coordinate system having a marker as an origin) and sets a virtual camera in the virtual space. For example, the virtual camera is arranged at a position in the virtual space corresponding to the position of the real camera in the real space. In addition, the mobile electronic device places a virtual object at a predetermined position (for example, the origin) in the virtual space. And based on a user's gesture input, while operating the virtual object of virtual space, the said virtual object is imaged with a virtual camera. The virtual space image thus obtained and the real space image captured by the outer camera 9 are superimposed and displayed on the display 2.

  In addition, for example, when an image including the marker illustrated in FIG. 21 and other specific objects is acquired from the outer camera 9, the mobile electronic device performs image recognition processing on the acquired image. Based on the result of the image recognition process, the portable electronic device displays a fish object and a fishing rod object (virtual object) superimposed on the real space image captured by the outer camera 9. At this time, when the user performs a gesture such as turning the reel of the fishing rod in the right side surface direction of the portable electronic device, the portable electronic device recognizes the gesture. Then, the portable electronic device performs a display on the display 2 as if a fish was caught.

  As described above, processing using the outer camera 9 and the infrared camera 4 and / or the distance measuring sensor 5 may be performed, and an image obtained by fusing the real world and the virtual space may be displayed on the display 2.

  In another application example, a game in which an image irradiated by the projector 8 and an image displayed on the display 2 are linked may be performed. FIG. 22 is a diagram illustrating an example of a game in which the projector 8 and the display 2 are linked.

  As shown in FIG. 22A, a batter is displayed on the display 2, and when the user performs a “par” gesture with the right hand, the portable electronic device 1 recognizes the gesture based on an image from the infrared camera 4. Then, the image of the ball is projected onto the right hand of the user using the projector 8. Next, when the user performs a gesture of waving his right hand, the image of the ball projected on the user's right hand disappears, the ball appears from the right end of the display 2, and the ball moves to the left of the screen (FIG. 22 ( B)). When the user presses the input button 6B at the timing when the ball reaches the predetermined range of the batter, the batter hits the ball back (FIG. 22C). The hit ball moves to the right of the screen. At a predetermined timing after the ball reaches the right end of the display 2 (a timing when the hit ball reaches the right hand), the user performs a “goo” gesture. Then, an image of the ball is projected onto the user's right hand as if the user caught the ball that was hit back (FIG. 22D). At this time, on the display 2, a character “OUT” indicating that the user has successfully caught the ball is displayed, a sound may be output, and the portable electronic device 1 may vibrate. The mobile electronic device 1 determines the position of the user's right hand based on the image from the infrared camera 4 and the distance measured by the distance measuring sensor 5 (the position on the plane parallel to the right side of the mobile electronic device and / or the right side). (Position in a direction perpendicular to the surface) may be recognized, and an image of the ball may be projected on the right hand of the user according to the recognition result. If the user does not perform the “goo” gesture at the predetermined timing or the timing at which the “goo” gesture deviates from the predetermined timing, the ball is not successfully caught and FIG. The ball image as in D) is not projected on the right hand. For example, an image different from the ball may be projected on the right hand, or characters “error” or “hit” may be displayed on the display 2. When the user's gesture cannot be recognized by the infrared camera 4, the same process as described above may be performed by detecting the hand movement using the distance measuring sensor 5.

  In another application example, the distance sensor 5 is used to calculate an absolute distance to an object (unspecified object), and an image is output to the display 2 based on the calculated distance, or a projector 8 is used. An image may be projected. For example, in the application example shown in FIG. 22, it is determined whether or not the ball has been successfully caught based on the absolute distance of the right hand when catching, and an image corresponding to the determination result is displayed on the display 2. Also good. In addition, the position of a specific object (position on the image) is detected from the image captured by the infrared camera 4, and the distance to the specific object is detected using the distance measuring sensor 5. A position in a three-dimensional space may be detected. Then, predetermined processing may be performed based on the position of the specific object in the three-dimensional space, and an image based on the result of the predetermined processing may be displayed on the display 2 or projected by the projector 8.

  In another application example, a distance (relative distance) of a specific object may be calculated based on an image captured by the infrared camera 4, and the vibrator 11 may be vibrated according to the calculated distance. For example, the vibrator 11 may be vibrated when the distance to the object is closer or farther than a predetermined threshold. Further, the absolute distance to a specific object may be calculated using the distance measuring sensor 5, and the vibrator 11 may be vibrated according to the calculated distance. Further, the movement of a specific object (for example, the movement of the right hand or the movement of another object) is detected using the infrared camera 4 and / or the distance measuring sensor 5, and the vibrator 11 is detected according to the movement of the specific object. May be vibrated. As a specific example, sound is output and the portable electronic device 1 is vibrated in response to the user moving his right hand to play a guitar. At this time, an image of a guitar string is displayed on the display 2, and the sound may change when the string displayed on the display 2 is pressed with a finger of the left hand. In such an application example, a louder sound is output as the user moves the right hand faster, and the portable electronic device 1 may vibrate more strongly.

  In the above description, a plurality of examples of processing using the infrared camera 4 and / or the distance measuring sensor 5 are shown, but other arbitrary processing may be performed. For example, a game that allows the user to perform a gesture at a predetermined timing may be performed in accordance with the content instructed by the display 2 or voice. In such a game, when a predetermined gesture is performed at a predetermined timing, the score is added, and the timing of the gesture performed by the user deviates from the predetermined timing or there is a gesture different from the predetermined gesture. If so, no score is added.

  As described above, the portable electronic device 1 of the present embodiment performs predetermined processing according to the type of application to be executed, and the infrared camera 4 and the distance measuring sensor 5 are used for the predetermined processing. For example, in an application, an image from the infrared camera 4 may be used for predetermined processing. In some applications, the distance calculated by the distance measuring sensor 5 is used for predetermined processing. In some applications, both the infrared camera 4 and the distance measuring sensor 5 are used for predetermined processing. For example, the infrared camera 4 is used when an application that detects a specific object (a predetermined object) and performs processing based on the detection result is executed. On the other hand, when an application that detects an unspecified object and performs processing based on the distance to the unspecified object is executed, the distance measuring sensor 5 is used.

  In one application, one of the infrared camera 4 and the distance measuring sensor 5 is selected according to the detection status, and the selected one is used for a predetermined process. For example, if a specific object cannot be detected in the image from the infrared camera 4 or the type of gesture cannot be determined, the distance measuring sensor 5 is selected instead of the infrared camera 4. Based on the information from the selected distance measuring sensor 5, the movement of the object is detected, and predetermined information processing is performed according to the detected movement. That is, among the infrared camera 4 and the distance measuring sensor 5, the infrared camera 4 is preferentially used for detecting an object. Then, an image is displayed on the display 2 or projected by the projector 8 based on the result of predetermined information processing. Of the infrared camera 4 and the distance measuring sensor 5, the distance measuring sensor 5 may be preferentially used for detecting an object.

  As another application example, when the user holds the portable electronic device 1 with both hands and is playing a game, when the user releases the right hand, the portable electronic device 1 is displayed on the display 2 instead of the game image. A menu screen may be displayed (or in addition). For example, the user ends the game being executed, saves the progress of the game, executes another game, or changes the setting of the mobile electronic device 1 by an operation using the menu screen. be able to. Specifically, the mobile electronic device 1 detects that the user has released the right hand using the infrared camera 4 or the distance measuring sensor 5. When detecting that the user has released the right hand, the portable electronic device 1 interrupts the game process being executed and displays the menu screen.

  Moreover, in the said portable electronic device 1, although the infrared camera 4 and the ranging sensor 5 etc. were arrange | positioned on the right side surface, when a user performs gesture input using a left hand, the portable electronic device 1 of the attitude | position shown in FIG. Rotate 180 degrees (that is, turn the portable electronic device 1 of FIG. 1 upside down). The portable electronic device 1 detects that the portable electronic device 1 has been rotated by detecting, for example, gravity using the posture detection unit 16. Then, the image displayed on the display 2 is rotated according to the detected attitude of the portable electronic device 1.

(Modification)
In addition, embodiment mentioned above is only an example, For example, the following modifications may be added.

  For example, the portable electronic device 1 is not limited to the above-described gesture input using the user's hand, and detects various other objects, performs predetermined processing according to the detection results of the objects, and results of the predetermined processing May be output to the display 2 or to the projector 8. For example, the input to the portable electronic device may be performed by imaging a specific object that exists in the lateral direction with the infrared camera 4. For example, when the user performs an operation on a specific object, the position and orientation of the specific object change. By imaging the specific object with the infrared camera 4, an operation performed on the specific object performed by the user can be detected. And the result of the process according to the said operation is output to the display 2 grade | etc.,. In this case, the portable electronic device 1 may detect both the user's hand and the object held by the user, or may detect only the object.

  Further, the mobile electronic device 1 may have any shape. For example, the mobile electronic device 1 (display 2) may be vertically long instead of horizontally long, or may be square. FIG. 23 is a diagram illustrating an example of the vertically long portable electronic device 50. FIG. 24 is a diagram illustrating an example of a horizontally long portable electronic device. In FIG. 23, the input button 6 is not displayed, but the portable electronic device 50 may include the input button 6. As shown in FIG. 23, the display 2 is provided on the front surface of the vertically long portable electronic device 50, and the infrared camera 4, the distance measuring sensor 5, and the projector 8 are provided on the right side surface. As described above, also in the portable electronic device 50, the user can perform gesture input with the right hand from the right side direction of the portable electronic device 50 using the infrared camera 4 and the distance measuring sensor 5 provided on the right side surface. it can.

  As described above, the portable electronic device of the present embodiment is provided with the infrared camera 4 and the like in the side surface direction (left and right direction) when the screen is viewed from the front regardless of whether the screen is vertically long or horizontally long. Here, “viewing the screen from the front” means viewing the screen so that a character string or an image displayed on the screen of the display 2 can be seen from a normal direction. For example, in FIG. 23, when the portable electronic device is held so that the screen is vertically long, the character string “ABC” and the image displayed on the screen can be seen from the normal direction, and this state is seen from the front. State. Also, for example, in FIG. 24, when the portable electronic device is held so that the screen is horizontally long, the character string “ABC” and the image displayed on the screen can be seen from the normal direction. It is in a state.

  As shown in FIGS. 23 and 24, the “side surface” in the present specification may include a bottom surface facing downward and an upper surface facing upward when the screen is viewed from the front. That is, in the portable electronic device of the present embodiment, the infrared camera 4, the distance measuring sensor 5, and the like may be provided on the “upper surface” facing upward or the “bottom surface” facing downward when the screen is viewed from the front. The infrared camera 4 and the distance measuring sensor 5 may be provided on the “left side” facing the left direction or the “right side” facing the right direction.

  Note that the portable electronic device of the present embodiment may detect the posture of its own device using the posture detection unit 16 and rotate characters, images, and the like displayed on the screen according to the detected posture. FIG. 25 is a diagram illustrating a state in which the portable electronic device is rotated 90 degrees to the right in the portable electronic device in which the infrared camera 4 and the distance measuring sensor 5 are provided on the side when the screen is viewed from the front. .

  As shown in FIG. 25, when the portable electronic device is held such that the screen is horizontally long, characters and images are visible on the screen from the normal direction, and the right side when the screen is viewed from the front. On the surface, an infrared camera 4 and a distance measuring sensor 5 are provided. On the other hand, when the portable electronic device is rotated 90 degrees to the right, the posture detection unit 16 detects gravity, for example, so that the portable electronic device detects the posture of the own device. The portable electronic device rotates the characters and images so that the characters and images displayed on the display 2 can be seen from the front according to the detection result. That is, even after the user has rotated 90 degrees to the right, the user can view the screen of the portable electronic device from the front. In this case, the infrared camera 4 and the distance measuring sensor 5 are positioned on the bottom surface when the screen of the portable electronic device is viewed from the front. Such portable electronic devices are also included in the scope of the present invention.

  That is, depending on how the portable electronic device is held, the infrared camera 4, the distance measuring sensor 5, etc. may be positioned downward or upward with respect to the screen. When the portable electronic device is held by a user in a certain posture, when the user views the screen from the front (when characters or images displayed on the screen are seen from a normal direction), the infrared camera 4 And the distance measuring sensor 5 and the like are positioned on the plane in the horizontal direction with respect to the screen. In such a posture, the user can perform gesture input to the mobile electronic device by performing a gesture with the right hand (or left hand) as described above from the right direction (or left direction) with respect to the screen. . Further, when the portable electronic device is held by the user in a different posture, when the user views the screen from the front, the infrared camera 4, the distance measuring sensor 5, etc. Come to be located. In such a posture, the user can make a gesture input to the portable electronic device by performing a gesture with the right hand (or left hand) as described above from above (or downward) with respect to the screen. .

  Further, for example, a configuration in which the infrared camera 4, the distance measuring sensor 5, and the projector 8 are provided on the top surface and the bottom surface as well as the side surface is also included in the scope of the present invention. In addition, the infrared camera 4 and the distance measuring sensor 5 may be provided on both the left side surface and the right side surface. In this case, for example, based on the distance measured by the distance measuring sensor, it may be determined which of the left side and the right side of the portable electronic device is held by the user. For example, the portable electronic device may determine that the side surface (gripping unit) corresponding to the shorter measured distance is gripped by the user.

  FIG. 26 is a diagram illustrating an example of a configuration in which the infrared camera 4 and the distance measuring sensor 5 are provided on both the side surface and the bottom surface. As shown in FIG. 26, the infrared camera 4 and the distance measuring sensor 5 are provided on the right side surface of the portable electronic device, and the infrared camera 4 and the distance measuring sensor 5 are provided on the bottom surface of the portable electronic device. The user can perform gesture input using the right hand while holding the portable electronic device with the left hand. In such a configuration, the infrared camera 4, the distance measuring sensor 5, etc. are positioned in the horizontal direction regardless of whether the portable electronic device is held horizontally or when the portable electronic device is held vertically. Therefore, the gesture input as described above can be performed in the horizontal direction of the screen.

  In this way, the user can perform gesture input from the side direction on the display screen with the right hand (or left hand) while holding the portable electronic device with the left hand (or right hand) and viewing the screen from the front. If it exists, the mobile electronic device may have any configuration.

  In the above embodiment, the infrared camera 4 and the distance measuring sensor 5 and the like are oriented in a direction perpendicular to the side surface (parallel to the screen). However, in other configurations, the infrared camera 4 and the distance measuring sensor 5 and the like are side surfaces. It may be provided so as to be inclined by a predetermined angle with respect to.

  FIG. 27 is a diagram of a portable electronic device viewed from the bottom when the infrared camera 4 and the distance measuring sensor 5 are tilted. As shown in FIG. 27, the imaging direction of the infrared camera 4 may be tilted by a predetermined angle in the screen direction instead of being perpendicular to the right side surface of the portable electronic device. Further, the detection direction of the distance measuring sensor 5 (light emission direction from the light source) may not be perpendicular to the right side surface of the portable electronic device but may be inclined by a predetermined angle in the screen direction. Similarly, the projector 8 may be tilted by a predetermined angle in the screen direction instead of being perpendicular to the right side surface of the portable electronic device. As described above, when the infrared camera 4, the distance measuring sensor 5, and the projector 8 are tilted in the screen direction, the user performs gesture input using a hand at a predetermined position in the screen direction rather than just beside the portable electronic device. It will be. The tilt of the infrared camera 4, the distance measuring sensor 5, and the projector 8 may be adjusted as appropriate so that the user can easily perform gesture input. For example, the infrared camera 4, the distance measuring sensor 5, and the projector 8 are tilted in the back direction. May be.

  Moreover, in the said embodiment, although the infrared camera 4 was used, instead of the infrared camera 4, the normal camera (camera which acquires RGB image) which images the image of visible light is used, and the said normal camera is used. The user's gesture may be recognized. Instead of the infrared camera 4, a camera that can capture both RGB images and infrared images may be used. FIG. 28 is a diagram illustrating a part of an image sensor capable of capturing both an RGB image and an infrared image. As shown in FIG. 28, each of the four elements (phototransistors) receives red light (R), green light (G), blue light (B), and infrared light (IR). By using an image sensor in which a large number of such combinations of four elements are arranged vertically and horizontally, an RGB image can be acquired and an infrared image can also be acquired. Thereby, a gesture can be recognized using either an RGB image or an infrared image. That is, an image suitable for gesture recognition can be selected from the acquired RGB image and infrared image. For example, when a specific object cannot be detected in the acquired infrared image due to the excessive amount of infrared light, the gesture can be recognized using the RGB image and vice versa.

  In the above embodiment, the projector 8 is provided. However, any projector may be used as long as the result of predetermined information processing (S13) according to the gesture input by the user is notified to the user by light. For example, instead of or in addition to the projector 8, a light source (for example, an LED, a halogen lamp, a fluorescent lamp, an EL lamp, etc.) that emits predetermined light may be provided in the portable electronic device 1.

  In the above embodiment, the image and the distance are separately acquired using the infrared camera 4 and the distance measuring sensor 5, but in another embodiment, for example, a distance image sensor of a TOF (Time of Flight) type is used. The distance and the distance for each pixel may be acquired using the distance image sensor. The portable electronic device detects a specific object based on the acquired image and distance, and outputs an image to the display 2 or the projector 8 according to the detection result.

  Further, in the above-described embodiment, the example in which the portable electronic device is held with one hand and the gesture input is performed on the portable electronic device with the other hand has been described. In another embodiment, the portable electronic device may be fixed to a user's arm (body), for example, a wristwatch-type device having a screen.

  The portable electronic device may have any shape, for example, a plate-like ellipse. For example, the portable electronic device may be foldable.

  In the above embodiment, a portable device is used. However, in another embodiment, a stationary device may be used.

DESCRIPTION OF SYMBOLS 1 Portable electronic device 2 Display 3 Touch panel 4 Infrared camera 5 Distance sensor 6 Input button 7 Irradiation part 8 Projector 9 Outer camera 11 Vibrator 16 Attitude detection part 17 GPS receiver 18 Geomagnetic sensor

Claims (8)

A handheld electronic device,
A left gripping part gripped by the user's left hand;
A right gripping part gripped by the right hand of the user;
A camera provided on a side surface of the electronic device and capable of imaging a space in the side surface direction;
Obtaining means for obtaining an input image captured by the camera;
Based on the input image acquired by the acquisition unit, the image is performed with the other hand on the opposite side to the one hand of the user holding the left grip part or the right grip part , which exists in the space in the side surface direction. Detecting means for detecting a gesture,
Irradiation means provided on the side surface and projecting an image according to the detection result by the detection means on the other hand ;
A display for displaying images,
Game processing means for performing predetermined game processing ,
The game processing means displays an image relating to a game on the display,
The detection means detects a gesture according to an image related to the game,
The game processing means performs the game processing based on the gesture detected by the detection means,
The illuminating means is an electronic apparatus that projects an image corresponding to the result of the game process onto the other hand .   The electronic apparatus according to claim 1, wherein a light irradiation direction by the irradiation unit is the same direction as an imaging direction of the camera. The irradiation means is a projector, an electronic device according to claim 1 or 2. The electronic device according to claim 1, wherein the irradiation unit continuously projects the image in accordance with the continuous gesture. The detecting means detects the position of the other hand ;
The irradiation means, on the basis of the detected position by the detecting means, for illuminating said image, the electronic device according to any one of claims 1 to 4. Wherein in accordance with a detection result by the detecting means comprises an audio output means for outputting a voice, electronic apparatus according to any one of claims 1 to 5. A handheld electronic device,
A left gripping part gripped by the user's left hand;
A right gripping part gripped by the right hand of the user;
A camera provided on a side surface of the electronic device and capable of imaging a space in the side surface direction;
Obtaining means for obtaining an input image captured by the camera;
Based on the input image acquired by the acquisition unit, the image is performed with the other hand on the opposite side to the one hand of the user holding the left grip part or the right grip part , which exists in the space in the side surface direction. Detecting means for detecting a gesture,
Irradiation means provided on the side surface and projecting an image according to the detection result by the detection means on the other hand ;
A display for displaying images,
An electronic apparatus comprising: interlocking means for interlocking an image projected by the irradiation means and an image displayed on the display . A handheld electronic device,
A left gripping part gripped by the user's left hand;
A right gripping part gripped by the right hand of the user;
A camera provided on a side surface of the electronic device and capable of imaging a space in the side surface direction;
Obtaining means for obtaining an input image captured by the camera;
Based on the input image acquired by the acquisition means, the movement of the other hand on the opposite side to the one hand of the user holding the left gripping part or the right gripping part existing in the space in the side surface direction is performed. Detecting means for detecting;
Irradiation means provided on the side surface and projecting an image according to the detection result by the detection means on the other hand ;
A display for displaying images,
An electronic apparatus comprising: interlocking means for interlocking an image projected by the irradiation means and an image displayed on the display .

Images