HK1173807B - Theme-based augmentation of photorepresentative view - Google Patents

Description

Theme-based enhancement of photorepresentative views

Technical Field

The invention relates to theme-based enhancement of photorepresentative views.

Background

Virtual reality systems exist for simulating a virtual environment in which a user may be immersed. A display (such as a heads-up display, head-mounted display, etc.) may be used to display the virtual environment. It may be desirable to provide the user with a virtual reality experience in which the user is not fully immersed, but rather remains connected to the real environment to some extent. Therefore, augmented reality systems are being developed to enhance a user's perception of a real-world environment with graphics, sound, and the like. As an example, an augmented reality system may be used to provide a cockpit simulation environment that integrates real objects with virtual images for flight training and simulation. Existing augmented reality techniques typically rely on predetermined data, such as a preconfigured model of the simulated environment, or the like, to cause the virtual elements to be properly integrated.

Disclosure of Invention

According to one aspect of the disclosure, a user's view of their environment is enhanced according to a theme selected by the user. Specifically, the display provides the user with a photo representation (photoresponsive) view of the user's point of view (variable point) of the user's environment. The display also displays enhancements associated with the user selected theme to enhance the view. Such enhanced displays are based on a spatial model of the physical environment that is generated and analyzed in real-time to identify features that may be enhanced. These identified features correspond to physical features in the physical environment.

This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that solve any or all disadvantages noted in any part of this disclosure.

Drawings

FIG. 1A illustrates an example use environment according to an embodiment of this disclosure.

FIG. 1B illustrates another example use environment according to an embodiment of the present disclosure.

Fig. 2 illustrates an example method of providing theme-based enhancements to a photorepresentative view in accordance with an embodiment of this disclosure.

FIG. 3 illustrates an exemplary schematic representation of forming a spatial model according to an embodiment of the present invention.

FIG. 4 illustrates exemplary theme-based enhancements according to an embodiment of the present invention.

FIG. 5 illustrates an example computing system according to an embodiment of this disclosure.

Detailed Description

Embodiments are disclosed herein that relate to providing theme-based enhancements to photorepresentative views. The photorepresentative view is provided by the display and is a fairly realistic (e.g., photo-like) representation of the view of the physical environment in which the user is located, the user's viewpoint. Enhancing the user's perception of their environment in this manner allows the user to experience a thematic view of their environment while still maintaining some degree of connectivity with the environment itself.

Aspects of the invention will now be described, by way of example, with reference to the above-listed embodiments shown. Components, process steps, and other elements that may be substantially the same in one or more embodiments are identified coordinately and are described with minimal repetition. It should be noted, however, that elements identified coordinately may also differ to some extent. It should also be noted that the drawings included in the present application are schematic and generally not drawn to scale. Rather, the various drawing scales, aspect ratios, and numbers of components shown in the figures may be purposely distorted to make particular features or relationships more visible.

FIG. 1A shows a schematic illustration of an example use environment (i.e., the physical environment 100 in which a user 102 is located) according to an embodiment of the present disclosure. It should be understood that FIG. 1A is for illustrative purposes and is not drawn to scale. Further, while physical environment 100 is shown in this example as a living room, it should be understood that such an environment is illustrative and, thus, is not intended to be limiting in any way. Rather, the physical environment 100 may be substantially any type of physical environment in which the user 102 is located, including but not limited to indoor environments, outdoor environments, familiar environments, unfamiliar environments, and the like. In addition to the user 102, the physical environment 100 includes several physical features 104, including a sofa 104a, a coffee table 104b, and a dog 104 c.

FIG. 1A also shows a display system 106 for providing theme-based enhancements via display output. The display system 106 includes a display 108 configured to provide a photo-representational view of the physical environment 100 of the viewpoint of the user 102. Non-limiting examples of display 108 include a heads-up display (HUD), a Head Mounted Display (HMD), and the like.

In some embodiments, the display 108 may be an optical see-through display having one or more sufficiently transparent portions. Such displays provide a photorepresentative view via a transparent portion through which the physical environment is visible to the user. However, in other embodiments, the display 108 may be an immersive display configured to provide a photorepresentative view by displaying a fully rendered image of a spatial model of the environment.

The display system 106 is configured to receive one or more user inputs (e.g., via an input device), including an input selecting a theme for enhancing the photorepresentative view. In response, display system 106 identifies physical features (e.g., objects, people, buildings, etc.) in the environment that can be enhanced according to the subject, such as physical features 104, and then displays such enhancements (e.g., while still providing at least some of the photorepresentative view). In this manner, the user 102 experiences a themed view of the physical environment 100 while still maintaining some degree of connectivity with the environment itself.

By way of example, the display 108 may display an image that the user 102 perceives as overlaying the physical features 104 to enhance the physical environment 100 according to the selected theme. As described in more detail below with reference to fig. 2, such a display system may utilize an image capture device (such as a depth camera) to obtain information about the physical environment 100, which the display system may utilize to generate a spatial model of the environment. The display system may then analyze the model to identify features within the model that may be enhanced.

It should be understood that the display system 106 as shown in FIG. 1A is not limiting. In other embodiments, one or more components of the display system may be implemented externally to provide a photorepresentative view at the display 108. Fig. 1B shows a schematic illustration of a user 102 in another example physical environment (i.e., physical environment 100B), where a display system 106B provides/transmits a photorepresentative view for viewing at a display 108. As described in more detail below with reference to fig. 3, such a display system may utilize an image capture system (such as a depth camera) to track the user 102 and the physical environment to generate a spatial model of the physical environment, e.g., the user's viewpoint.

The display system (e.g., display system 106B) may be configured to provide theme-based enhancements in any suitable manner. FIG. 2 illustrates an example method 200 of providing theme-based enhancements to a photorepresentative view. Such a method may be provided by a display system (e.g., display system 106B), for example, by executing instructions stored thereon.

At 202, method 200 includes receiving input from a user selecting a theme for use in enhancing the photorepresentative view. Such input may be received from an input device configured to communicate with the display system, such as user-selectable buttons, virtual buttons, or other user interfaces displayed on a display (e.g., display 108) or at another location, and so forth.

At 204, the method 200 includes: environmental information of the physical environment is obtained optically and in real time. Such environment information may be any suitable information describing the physical environment and its characteristics, such as physical characteristics 104 of physical environment 100 shown in FIG. 1A. Examples of suitable environmental information include, but are not limited to, image data (e.g., color information, depth information, infrared information, etc.), size data, surface data, motion data, audio data, and the like.

The display system (e.g., display system 106B) may be configured to obtain environmental information of the physical environment (e.g., physical environment 100B) in any suitable manner. As a non-limiting example, the display system may further include one or more sensors configured to obtain such environmental information optically and in real-time. As such, method 200 may include detecting the environmental information via one or more sensors associated with the display, as indicated at 206.

In some embodiments, such sensors may be located near the display 108 to capture environmental information from the viewpoint of the user 102. FIG. 1A illustrates such an example sensor 110 associated with the display 108 and positioned proximate to the display 108. However, as another example, FIG. 1B shows an example sensor 110B associated with the display 108 and positioned at a distance from the display 108. In the latter case, the display system 106B may be configured to determine the viewpoint of the user 102 based on the detected information. As a non-limiting example, the sensor may include a capture device, such as a depth camera that visually monitors or tracks the observed scene of the physical environment and its features. By tracking the position and orientation, etc., of the user 102, the display system 106B may communicate information to the display 108 such that it provides a photorepresentative view that reflects the user's current and/or changing viewpoint.

At 208, method 200 includes generating a spatial model of the physical environment based on the environmental information. Such a spatial model may be any suitable model for representing the physical environment of the user's viewpoint, such as a two-dimensional model, a three-dimensional model, or the like. The spatial model may include a representation of one or more physical features within a photorepresentative view of the physical environment from the user's viewpoint. In particular, the spatial model includes a representation of the physical environment and a representation of objects within the physical environment. The spatial model is thus indicative of the spatial positioning of the objects within the environment, as well as the relative positioning to each other. By way of non-limiting example, the spatial model may be a computer-generated model. Further, the model may be dynamically updated in real-time to account for changing user vantage points.

Fig. 3 shows an example schematic illustration of forming a spatial model for the case of a display system employing depth image analysis. Although FIG. 3, in which sensor 110 detects observed scene 112, is discussed in more detail below with reference to FIG. 1A, it should be understood that this discussion is also applicable to FIG. 1B, in which sensor 110B detects observed scene 112B. In the former case, the sensor 110 is placed near the display 108, and thus the obtained context information is already from the user's viewpoint (e.g., the sensor is aimed due to movement of the user's head). In the latter case, because sensor 110B is placed at a distance from display 108, the display system may determine the user's point of view (e.g., body position/orientation, line of sight, etc.) from the environmental information to generate a spatial model of the physical environment from the user's perspective.

Continuing with fig. 3, the depth image analysis system of display system 106 may utilize a sensor 110 (e.g., a depth camera) to visually monitor or track physical features (e.g., objects, people, etc.) of physical environment 100 within an observed scene 112 (as observed by sensor 110). In the illustrated example, a physical feature in the form of a dog 104c is tracked by the sensor 110. It should be understood that the scenario illustrated in FIG. 3 is provided as an example, but is not meant to be limiting in any way. Rather, the illustrated scenario is intended to demonstrate a general concept that may be applied to a wide variety of different applications without departing from the scope of the present disclosure. As such, other physical features within the observed scene 112 may additionally or alternatively be tracked by the sensor 110.

FIG. 3 illustrates a simplified processing pipeline in which dog 104c in observed scene 112 is modeled as a virtual skeleton 338 that may be used to generate virtual avatar 316. By modeling each of the physical features within the environment in this manner, a spatial model of the environment may be generated. Accordingly, avatars, virtual objects, surfaces, floors, etc. are suitably parameterized within the spatial model to accurately represent tracked positions, orientations, movements, etc. of their physical counterparts, thereby producing a spatial model that provides an accurate representation of the physical environment. It is to be understood that the processing pipeline may include additional steps and/or alternative steps than those depicted in fig. 3 without departing from the scope of the present invention.

As shown in fig. 3, the simplified processing pipeline begins with a target (e.g., dog 104c) being imaged by a capture device such as sensor 110 (e.g., depth camera). It should be understood that several objects within the environment, as well as the environment itself (e.g., walls, floors, etc.), may also be imaged. The depth camera may determine, for each pixel, a depth of a surface in the observed scene relative to the depth camera. Substantially any depth-finding (depth-finding) technique may be used without departing from the scope of the present disclosure. Example depth finding techniques are discussed in more detail with reference to capture device 518 of FIG. 5.

The depth information determined for each pixel may be used to generate a depth map 336. Such a depth map may take the form of substantially any suitable data structure, including but not limited to a matrix including depth values for each pixel of the observed scene. In fig. 3, the depth map 336 is schematically shown as a pixelated grid of the contour of the object. This illustration is for the sake of clarity of understanding and not for the purpose of technical accuracy. It will be appreciated that the depth map typically includes depth information for all pixels (not just the pixels that image the object) and that the viewpoint of the sensor 110 does not result in the contours depicted in fig. 3.

In some embodiments, virtual skeleton 338 may be derived from depth map 336 to provide a machine-readable representation of the target. In other words, virtual skeleton 338 may be derived from depth map 336 to model the object. Virtual skeleton 338 may be derived from the depth map in any suitable manner. In some embodiments, one or more skeleton-fitting algorithms may be applied to the depth map. The present invention is compatible with substantially any skeletal modeling technique.

Virtual skeleton 338 may include a plurality of joints, each joint corresponding to a portion of a target. In FIG. 3, virtual skeleton 338 is shown as a multi-joint line drawing. This illustration is for the sake of clarity of understanding and not for the purpose of technical accuracy. A virtual skeleton according to the present invention may include substantially any number of joints, each of which may be associated with substantially any number of parameters (e.g., three-dimensional joint positions, joint rotations, body positions of corresponding body parts (e.g., hand open, hand closed, etc.). It should be appreciated that the virtual skeleton may take the form of a data structure as follows: the data structure includes one or more parameters for each of a plurality of skeletal joints (e.g., a joint matrix containing an x-position, a y-position, a z-position, and a rotation for each joint). In some embodiments, other types of virtual skeletons may be used (e.g., a wireframe, a set of shape primitives, etc.).

As shown in FIG. 3, a virtual avatar 316 may be generated. Such avatars provide virtual representations of targets within a spatial model corresponding to the environment. More specifically, because virtual skeleton 338 models the tracked target, and a virtual avatar is generated based on virtual skeleton 338, virtual avatar 316 serves as a digital representation of the target within the spatial model. It should be appreciated that by generating the spatial model in this manner, any one or more portions of the spatial model may optionally be rendered for display. Furthermore, such spatial models support enhancements in that the properties of the digital representations within the model can be modified and rendered for display.

It should be appreciated that although the above-described techniques for modeling objects within a physical environment are directed to modeling a single target, such description is non-limiting. Thus, several objectives may be modeled using the techniques described above without departing from the scope of the present disclosure. Also, as noted above, physical features other than moving objects may be modeled, such as floors, walls, ceilings, and so forth.

In this way, by modeling the physical environment and its features, a spatial model of the physical environment may be generated. Since the environmental information can be obtained optically and in real time, not only can the spatial model be generated and/or updated in real time, but the spatial model does not have to rely on preconfigured data. Thus, unlike traditional simulation environments, users may utilize such display systems in "new environments" for which the display systems have little or no pre-existing knowledge (e.g., pre-configured spatial models, GPS data, etc.). As such, the user may experience theme-based enhancements for a variety of different physical environments, including new environments for which the system does not have prior information.

Returning to FIG. 2, after generating the spatial model, method 200 next includes, at 210, identifying, via analysis of the spatial model, one or more features within the spatial model, each feature corresponding to one or more physical features in the physical environment. In this way, the display system determines features within the user's view of the environment. Such analysis may include any suitable analysis, such as object recognition, gesture recognition, facial recognition, voice recognition, audio recognition, and so forth. In some cases, such analysis may generate a general description (e.g., indicating the size of the object), etc., while in other cases, such analysis may generate a more detailed description (e.g., indicating that the object is, for example, a dog).

Continuing with FIG. 2, at 212, method 200 next includes, based on the analysis, displaying one or more enhancements to the one or more identified features on a display (e.g., display 108). In some embodiments, the augmentation may be displayed while still providing at least a portion of the photorepresentative view of the physical environment, for example. The augmentation is associated with a theme selected by the user. In some embodiments, the enhancement may be selected from a plurality of enhancements associated with the topic, and such selection is based on features identified within the spatial model, as indicated at 214. For example, if a large object is identified within the spatial model, an enhancement of similar magnitude in scale may be selected from the available enhancements associated with the subject. For example, a relatively large object in a living room, such as a sofa, may result in the use of certain available enhancements within a theme (e.g., a large castle in a mid-century theme). Assuming a natural/forest theme, tall, narrow objects in the physical environment may be covered with a virtual tree.

The displayed augmentation may be substantially any type of virtual image that augments the photorepresentative view provided to the user. As an example, an image may be displayed that overlays the corresponding physical feature in the physical environment from the user's viewpoint, as indicated at 216. By way of non-limiting example, virtual skin may be applied to the identified features within the spatial model, and an image of the virtual skin may be displayed. Because the skin is applied to features within the model, the displayed skin will appear to overlay the corresponding physical features in the environment, according to the user.

FIG. 4 shows an example of the user 102 of FIG. 1A, wherein the user has selected a century theme. In this example, the display system 106 has generated a spatial model of the physical environment 100 and the identified features (i.e., the couch 104a, the coffee table 104b, and the dog 104 c). Based on this analysis, an enhancement of the photorepresentative view is displayed at the display 108, shown in the expanded view at 400. Displaying this enhancement enables the user to see a castle 402 overlaid onto the sofa 104a, a green plant 404 overlaid onto the coffee table 104b, and apparel 406 overlaid onto the dog 104 c. Because the display of the augmentation is based on an analysis of the spatial model, the augmentation can be customized to the physical environment. For example, the castellations 402 may be customized to correspond with the dimensions and features of the sofa 104 a. Furthermore, because the spatial model may be updated in real-time as new environmental information is obtained, the augmented display may also be updated in real-time. Thus, as the user moves through the physical environment, an enhanced display may be maintained, allowing the user to continue the theme-based experience. As such, the enhanced display is not "static" in any sense. Conversely, as the user moves, the display of the augmentation may be dynamically adjusted such that the augmentation continues to be displayed according to the dynamically changing viewpoint of the user.

It should be understood that the scenario illustrated in FIG. 4 is provided as an example, but is not meant to be limiting in any way. Rather, the illustrated scenario is intended to demonstrate a general concept that may be applied to a wide variety of different applications without departing from the scope of the present disclosure.

Continuing with FIG. 2, the enhancement may be displayed in any suitable manner. For example, in the case of an optical see-through display, the user may have been viewing the environment via a photorepresentative view provided by an optically transparent portion of the display. In this case, one or more images of the virtual object may be displayed on the display, the images overlaying corresponding physical features in the physical environment from the user's viewpoint. As such, the displayed image enhances the user's view, as indicated at 218. In this way, users are provided with a theme-based experience of their environment while still remaining somewhat connected to the environment itself.

Alternatively, in the case of an immersive display, the user may have been viewing the environment via a photorepresentative view provided via a fully rendered image of the spatial model (e.g., the entire display is rendered, rather than the user viewing the scene directly through an optically transparent portion of the display). In this case, one or more features identified within the spatial model may be modified, and in response, a fully rendered image of the spatial model may be displayed that reflects such modification of the spatial model. Because the features identified within the spatial model may be sufficiently maintained, the user may perceive the modification as covering physical features in the environment, as indicated at 220.

Thus, whether the display is an optical see-through display or an immersive display, the user is provided with a theme-based experience of their environment while still maintaining some degree of association with the environment itself. In other words, typically some portion of the displayed content remains a photo representation, e.g., an accurate view of the physical environment is provided with a relatively high level of fidelity, such as photo-like. That is, in some cases, the entire physical environment may be overlaid, skin changed (skin), etc. with appropriate theme-based modifications selected in response to the spatial model.

Furthermore, because the display system obtains environmental information optically and in real-time, the enhanced display may be maintained as the user moves through the physical environment, as indicated at 222. In other words, the augmentation may be dynamically displayed as the user moves through the physical environment to produce a corresponding change in the photorepresentative view. In some embodiments, this may include modifying the augmentation and/or spatial model such that the displayed augmentation continues to be displayed according to the user's viewpoint, even though the user's viewpoint also changes in real-time. Thus, the user may naturally continue the theme-based experience as the user moves through and/or interacts with the environment.

In some embodiments, the methods and processes described above may be bundled into a computing system comprising one or more computers. In particular, the methods and processes described herein may be implemented as a computer application, computer service, computer API, computer library, and/or other computer program product.

FIG. 5 schematically illustrates a non-limiting computing system 500 that can perform one or more of the above-described methods and processes. Computing system 500 is shown in simplified form. It should be understood that substantially any computer architecture may be used without departing from the scope of this disclosure. In different embodiments, the computing system 500 may take the form of a mainframe computer, server computer, desktop computer, laptop computer, tablet computer, home entertainment computer, network computing device, mobile communication device, gaming device, or the like.

Computing system 500 includes a logic subsystem 502 and a data-holding subsystem 504. Computing system 500 may optionally include a display subsystem 506, a communication subsystem 508, an environment subsystem 510, an analysis subsystem 512, and/or other components not shown in fig. 5. Computing system 500 may also optionally include user input devices such as the following: such as a keyboard, mouse, game controller, camera, microphone, and/or touch screen, etc.

Logic subsystem 502 may include one or more physical devices configured to execute one or more instructions. For example, the logic subsystem may be configured to execute one or more instructions that are part of one or more applications, services, programs, routines, libraries, objects, components, data structures, or other logical constructs. Such instructions may be implemented to perform a task, implement a data type, transform the state of one or more devices, or otherwise arrive at a desired result.

The logic subsystem may include one or more processors configured to execute software instructions. Additionally or alternatively, the logic subsystem may include one or more hardware or firmware logic machines configured to execute hardware or firmware instructions. The processors of the logic subsystem may be single-core or multi-core, and the programs executing thereon may be configured for parallel or distributed processing. The logic subsystem may optionally include individual components that are distributed throughout two or more devices, which may be remotely located and/or configured for coordinated processing. One or more aspects of the logic subsystem may be virtualized and executed by remotely accessible networked computing devices configured in a cloud computing configuration.

Data-holding subsystem 504 may be operatively coupled to the display and one or more sensors (e.g., sensors 516), and may include one or more physical, non-transitory devices configured to hold data and/or instructions executable by the logic subsystem to implement the methods and processes described herein. In implementing such methods and processes, the state of data-holding subsystem 504 may be transformed (e.g., to hold different data).

Data-holding subsystem 504 may include removable media and/or built-in devices. Data-holding subsystem 504 may include optical memory devices (e.g., CD, DVD, HD-DVD, Blu-ray disc, etc.), semiconductor memory devices (e.g., RAM, EPROM, EEPROM, etc.) and/or magnetic memory devices (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.), among others. Data-holding subsystem 504 may include devices with one or more of the following characteristics: volatile, nonvolatile, dynamic, static, read/write, read-only, random access, sequential access, location addressable, file addressable, and content addressable. In some embodiments, logic subsystem 502 and data-holding subsystem 504 may be integrated into one or more common devices, such as an application specific integrated circuit or a system on a chip.

FIG. 5 also illustrates an aspect of the data-holding subsystem in the form of removable computer-readable storage media 514, which may be used to store and/or transfer data and/or instructions executable to implement the herein described methods and processes. Removable computer-readable storage medium 514 may take the form of, inter alia, a CD, DVD, HD-DVD, blu-ray disc, EEPROM, and/or floppy disk.

As will be appreciated, data-holding subsystem 504 includes one or more physical, non-transitory devices. Rather, in some embodiments, aspects of the instructions described herein may propagate in a transient manner through a pure signal (e.g., an electromagnetic signal, an optical signal, etc.) that is not held by the physical device for at least a finite duration. In addition, data and/or other forms of information pertaining to the present invention may propagate through a pure signal.

When included, display subsystem 506 may be used to present a visual representation (e.g., an avatar and/or a three-dimensional virtual world) of the data held by data-holding subsystem 504. As the herein described methods and processes change the data held by the data-holding subsystem, and thus transform the state of the data-holding subsystem, the state of display subsystem 506 may likewise be transformed to visually represent changes in the underlying data. For example, computing system 500 may be configured to present a driving game for display on a display device of display subsystem 506. As such, computing system 500 may include display output to output a driving game interface to a display device. Display subsystem 506 may include one or more display devices using virtually any type of technology. Such display devices may be combined with logic subsystem 502 and/or data-holding subsystem 504 in a shared enclosure, or such display devices may be peripheral display devices connected to the logic subsystem via a display output.

When included, the communication subsystem 508 may be configured to communicatively couple the computing system 500 with one or more other computing devices. The communication subsystem may include wired and/or wireless communication devices compatible with one or more different communication protocols. By way of non-limiting example, the communication subsystem may be configured to communicate via a wireless telephony network, a wireless local area network, a wired local area network, a wireless wide area network, a wired wide area network, or the like. In some embodiments, the communication subsystem may allow computing system 500 to send and/or receive messages to and/or from other devices via a network such as the internet.

The environmental subsystem 510 may include one or more sensors 516 for obtaining environmental information optically and in real-time. The sensors 516 may include an integrated and/or peripheral capture device 518 configured to obtain depth images of one or more targets. In either case, computing system 500 may include peripheral inputs to receive depth images from the depth camera and pass the received depth images to the logic subsystem for processing. Capture device 518 may be configured to capture video with depth information via any suitable technique (e.g., time-of-flight, structured light, stereo image, etc.). As such, capture device 518 may include a depth camera, a video camera, a stereo camera, and/or other suitable capture devices.

For example, in time-of-flight analysis, the capture device 518 may emit infrared light toward the target and then use a sensor to detect the backscattered light from the surface of the target. In some cases, pulsed infrared light may be used, where the time between an outgoing light pulse and a corresponding incoming light pulse may be measured and used to determine a physical distance from the capture device to a particular location on the target. In some cases, the phase of the outgoing light wave may be compared to the phase of the incoming light wave to determine a phase shift, and the phase shift may be used to determine a physical distance from the capture device to a particular location on the target.

In another example, time-of-flight analysis may be used to indirectly determine a physical distance from the capture device to a particular location on the target by analyzing the intensity of the reflected beam of light over time via techniques such as shuttered light pulse imaging.

In another example, the capture device 518 may utilize structured light analysis to capture depth information. In such an analysis, patterned light (e.g., light displayed as a known pattern such as a grid pattern, a stripe pattern, or constellation points) may be projected onto a target. After landing on the surface of the target, the pattern may become distorted, and such distortion of the pattern may be studied to determine a physical distance from the capture device to a location on the target.

In another example, a capture device may include two or more physically separated cameras that view a target from different angles to obtain visual stereo data. In this case, the visual stereo data may be parsed to generate a depth image.

In other embodiments, capture device 518 may utilize other techniques to measure and/or calculate depth values. Further, the capture device 518 may organize the calculated depth information into "Z layers," e.g., layers perpendicular to a Z axis extending from the depth camera along its line of sight to the target.

In some embodiments, two or more cameras may be integrated into one integrated capture device. For example, a depth camera and a video camera (e.g., an RGB video camera) may be integrated into a common capture device. In some embodiments, two or more separate capture devices may be used in conjunction. For example, a depth camera and a separate camera may be used. When a camera is used, the camera may be used to provide: target tracking data, confirmation data to correct for target tracking, image capture, facial recognition, high precision tracking of a finger (or other small feature), light sensing, and/or other functions.

The environment subsystem 510 may be further configured to generate a spatial model of the physical environment based on the environmental information. It is to be understood that at least some target analysis and tracking operations may be performed by the logic machine of one or more capture devices. The capture device may include one or more onboard processing units configured to perform one or more target analysis and/or tracking functions. The capture device may include firmware to help update such on-board processing logic.

Computing system 500 may optionally include one or more input devices, such as input device 520. The input device 520 may be used to receive user input, for example, selecting a theme for enhancing the photorepresentative view. The input device may be used to control the operation of the computing system. In the context of gaming, input devices may be used to control aspects of a game that are not controlled by the target recognition, tracking, and analysis methods and processes described herein. In some embodiments, the input devices may include one or more of accelerometers, gyroscopes, infrared target/sensor systems, etc., which may be used to measure movement of the controllers in physical space. In some embodiments, the computing system may optionally include and/or utilize input gloves, keyboards, mice, trackpads, trackballs, touch screens, buttons, switches, dials, and/or other input devices. As will be appreciated, target recognition, tracking, and analysis may be used to control or augment aspects of a game or other application that are conventionally controlled by an input device, such as a game controller. In some embodiments, the target tracking described herein may be used as a complete replacement for other forms of user input, while in other embodiments, such target tracking may be used to supplement one or more other forms of user input.

The analysis subsystem 512 may be configured to then analyze the spatial model created by the environmental subsystem 510. Such analysis may include object recognition, gesture recognition, facial recognition, voice recognition, audio recognition, and/or any other suitable type of analysis.

It will be appreciated that the configurations and/or approaches described herein are exemplary in nature, and that these specific embodiments or examples are not to be considered in a limiting sense, because numerous variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. As such, various acts illustrated may be performed in the sequence illustrated, in other sequences, in parallel, or in some cases omitted. Also, the order of the above-described processes may be changed.

The subject matter of the present disclosure includes all novel and nonobvious combinations and subcombinations of the various processes, systems and configurations, and other features, functions, acts, and/or properties disclosed herein, as well as any and all equivalents thereof.

Claims (9)

1. A method (200) of providing a theme-based enhancement of a photorepresentative view of a user's viewpoint on a display configured to provide the photorepresentative view of a physical environment in which the user is located, the method comprising:

receiving (202) input from the user selecting a theme for use in augmenting the photorepresentative view;

obtaining (204) environment information of the physical environment optically and in real time;

generating (208) a spatial model of the physical environment based on the environment information;

identifying (210), via analysis of the spatial model, one or more features within the spatial model, each feature corresponding to one or more physical features in the physical environment;

after analyzing the spatial model, selecting the enhancement from a plurality of enhancements associated with the topic, the selecting based on the size of the identified one or more features and/or what the identified one or more features are; and

displaying (212) an augmentation of the identified feature on the display, the augmentation being associated with the theme.

2. The method of claim 1, wherein displaying the augmentation comprises displaying an image of the user viewpoint that overlays a physical feature in the physical environment corresponding to the identified feature.

3. The method of claim 1, wherein the display is an optical see-through display configured to provide the photorepresentative view via one or more sufficiently transparent portions of the display through which the physical environment is visible to the user.

4. The method of claim 3, wherein displaying the augmentation comprises overlaying a virtual object onto the identified feature of the spatial model, and displaying an image of the virtual object, wherein the image of the virtual object overlays a corresponding physical feature in the physical environment visible to the user through the display from the viewpoint of the user.

5. The method of claim 1, wherein the display is an immersive display configured to provide the photorepresentative view by displaying a fully rendered image of the spatial model.

6. The method of claim 5, wherein displaying the augmentation comprises modifying one or more features identified within the spatial model, and in response, displaying a fully rendered image of the spatial model that reflects such modification to the spatial model.

7. The method of claim 1, further comprising maintaining the augmented display as the user moves through the physical environment.

8. A display system (500) for providing theme-based enhancements via display output, comprising:

a display (506) configured to provide a photorepresentative view of a user's viewpoint of a physical environment in which the user is located;

one or more sensors (516) configured to obtain environmental information of the physical environment optically and in real-time;

a data-holding subsystem (504) operably coupled with the display and the one or more sensors, the data-holding subsystem containing instructions executable by a logic subsystem to:

receiving (202) input from the user selecting a theme for use in augmenting the photorepresentative view;

generating (208) a spatial model of the physical environment based on the environment information;

identifying (210), via analysis of the spatial model, one or more features within the spatial model, each feature corresponding to one or more physical features in the physical environment;

after analyzing the spatial model, selecting the enhancement from a plurality of enhancements associated with the topic, the selecting based on the size of the identified one or more features and/or what the identified one or more features are; and

displaying (2121) an augmentation of the identified feature on the display, the augmentation being associated with the theme.

9. The display system of claim 8, wherein the one or more sensors comprise an image capture device configured to obtain one or more depth images of the physical environment.