CN110249291A - System and method for augmented reality content delivery in a pre-capture environment - Google Patents

Abstract

The present disclosure describes systems and methods for Augmented Reality (AR) content delivery in a pre-capture environment. In one embodiment, the AR client collects information about the physical environment with the help of sensors embedded in the AR headset. The client reconstructs the 3D model of the physical environment from the collected information and sends it to the AR content server. Based on the 3D reconstruction information obtained at the AR content server, the server modifies the content. The AR content server compares the AR content with the 3D reconstruction data and performs a visibility analysis of the AR content. Based on the visibility analysis, AR elements occluded by real objects are removed to reduce the data size of the AR content. Finally, the AR content server streams the modified content to the AR client and the AR client renders the content and sends it to the headset.

Description

Systems and methods for augmented reality content delivery in a pre-capture environment

Cross References to Related Applications

This application is a U.S. Provisional Patent Application entitled "System and Method for AugmentedReality Content Delivery in Pre-Captured Environments" filed on February 1, 2017 Formal application Serial No. 62/453,317 and claiming the benefit thereof under 35 U.S.C. § 119(e), the Provisional Patent Application is hereby incorporated by reference in its entirety.

Background technique

Storing digital media in the cloud enables accessibility from anywhere in the world, as long as the client has an Internet connection. Digital media with greater levels of realism are encoded using high-resolution formats that require large file sizes. Transmitting this information (eg, streaming the digital media to a client device) requires a commensurately large allocation of communication resources. Visually rich virtual reality (VR) content and augmented reality (AR) content both consume large amounts of data, which is problematic for AR and VR content delivery. The limited bandwidth of the data connection between the service delivering the content and the clients consuming the content is a significant bottleneck.

For example, the spherical video used by cinematic VR technology dramatically increases the video resolution requirements for online video feeds. In cinematic VR, the delivered video stream contains not only all the pixels as seen on a screen like normal 2D video, but also pixels from a full 360° field of view from a single point of view. When cinematic VR content is consumed, only a small area of the entire video content is visible at any given point in time. The smaller cropped area of the spherical view displayed on the client device needs to provide full HD (ie, 1080p) resolution in order for the viewing experience to match the resolution provided by high resolution 2D video in use today. When comparing monoscopic 360 degree spherical video to stereoscopic 360 degree 3D spherical video (where separate image feeds are required for each eye), the bandwidth required for content delivery becomes a major bottleneck. Next-generation immersive content formats, which not only provide a 360-degree stereoscopic view from a single point of view, but also allow users to move within a restricted area within the content, will consume an exponential increase in information and communication resources.

The ability to move within VR space is not the only driver behind the size of these next-generation formats, as next-generation AR display technologies will preferably represent the same level of spatial realism. Current content streaming methods for delivering AR and VR experiences rely on traditional data transmission optimization techniques (eg, spatial 2D image pixel compression and temporal motion compensation). Traditional data compression techniques do not address the nuances of AR use cases and can be improved upon.

Contents of the invention

This application describes systems and methods for selectively providing augmented reality (AR) information to AR display client devices. In some implementations, the bandwidth used to provide AR information to the client device may be reduced by selectively delivering to the client device only AR content elements that are not outwardly obscured by physical objects in the AR viewing position.

For example, in one embodiment, a system (eg, an AR content server) determines the location of an AR display device within an AR viewing environment. For at least a first element of AR content (eg, a video or an AR object), the system selects a display location within the AR viewing environment. The system determines whether any physical object in the AR viewing environment is located along a line of sight between the AR display device and the display location of the first element. The system transmits the first element of AR content to the AR display device only in response to determining that no physical object is in the line of sight. The system can handle some elements of AR content in different display positions (including elements that may be in motion and thus have variable display positions), and can send to the AR display device only the position from the perspective of the AR display device without being manipulated by physical objects. Blocking elements. In some implementations, the AR element may be a portion of an AR object or video, whereby the unoccluded portion of the object or video is sent to the client and the occluded portion is not.

An AR headset may perform an exemplary method according to an embodiment. The method includes acquiring a digitally reconstructed three-dimensional (3D) environment of an augmented reality (AR) viewing location, detecting an object in the digitally reconstructed 3D environment, determining the depth and geometry of the object, transmitting content specific to the augmented reality (AR) A stream request is made to the server, the request including information indicative of the depth and the geometry of the object, and the AR content stream is received from the server, the AR content stream not including AR content occluded by the object.

According to an embodiment, the method further includes determining the AR viewing location using one or more of Global Positioning System (GPS) data, wireless data, and image recognition. In one embodiment, the digitally reconstructed 3D environment is obtained via one or more of stored previous AR sessions, a combination of stored data and real-time collected data, or a point cloud of depth data.

In another embodiment, the digitally reconstructed 3D environment is acquired by scanning the AR viewing location by examining the AR viewing location from multiple viewpoints.

In one embodiment, the method further comprises tracking viewpoint position and orientation within the digitally reconstructed 3D environment.

In one embodiment, said information indicative of said depth and said geometry of said object comprises one or more of real-time raw sensor data, a depth map of said object, or an RGB-D data stream.

In one embodiment, the type of AR content requested is immersive video and said information indicative of said depth and said geometry of at least one real world object is formatted as a spherical depth of field for a real world AR viewing position picture.

Another embodiment relates to an augmented reality (AR) content server including a processor, a communication interface, and data storage. In this embodiment, the AR content server is configured to receive a request for AR content via the communication interface, retrieve the requested AR content from the data store, obtain a digitally reconstructed three-dimensional (3D) view of the AR viewing position. environment, receiving a viewpoint position and orientation within an AR viewing position via the communication interface, performing visibility analysis via a comparison of the requested AR content with the digitally reconstructed 3D environment and viewpoint position and orientation, and by The visibility analysis modifies the requested AR content by removing data associated with one or more objects not visible from the viewpoint position and orientation.

According to one embodiment, said digitally reconstructed 3D environment is retrievable from one or more of said data store, augmented reality (AR client) or a combination of said data store and said AR client.

In another embodiment, or in the same embodiment, said modification of the requested AR content can be a function of the content type of the requested AR content, where the content type includes three-dimensional (3D) scenes, light fields One or more of , immersive video with depth data.

Additionally, in one or more implementations, the modified requested AR content can include a previously removed three-dimensional (3D) object inserted within the requested AR content based on the updated viewpoint position and orientation.

In one embodiment, the visibility analysis includes identifying where in the digitally reconstructed 3D environment has a depth value that is smaller than depth data associated with the requested AR content.

Another embodiment relates to a method comprising receiving a request for an augmented reality (AR) content video stream, retrieving the requested AR content video stream from a data store, obtaining a digitally reconstructed 3D environment of a real-world AR viewing location, receiving the analyzing the digitally reconstructed 3D environment and the viewpoint position and orientation to determine whether the requested AR content video stream is included in the digitally reconstructed 3D environment one or more objects that are occluded, and removing the one or more objects that are occluded in the digitally reconstructed 3D environment.

In one embodiment, the method includes sending the modified AR content video stream to the AR display device.

In one embodiment, said analyzing said digitally reconstructed 3D environment and said viewpoint position and orientation comprises determining the position of said AR display device, identifying and one or more physical objects facing the determined line of sight, and based on the line of sight, determining whether the one or more objects in the AR content stream are occluded by the one or more physical objects.

In one embodiment, said analyzing said digitally reconstructed 3D environment and said viewpoint position and orientation comprises comparing a depth value of said digitally reconstructed 3D environment with a depth value in said AR content video stream, and Discarding the part of the AR content video stream whose depth value is greater than the corresponding depth value in the digitally reconstructed 3D environment, to compensate for the loss from the digitally reconstructed 3D environment caused by one or more dynamic objects and/or static objects of occlusion.

In a variant of the embodiment of the present application, the client knows the type of content it requests from the server, and based on this, can provide context information in as compact a format as possible. For example, for immersive video, the client may not send the entire environment model, but only a spherical depth map of the environment presented from this moment using the viewpoint.

In a variation of the embodiments of the present application, the AR client device does not perform viewpoint and orientation tracking, but uses an external outside-looking-in tracking scheme. In a first further variant, the AR client device first receives tracking information from an external tracking solution, transforms it into the same coordinate system as used in the 3D reconstruction, and then sends it to the content server. In a second further variant, the external tracking scheme transforms the viewpoint and orientation into the same coordinate system as used in the 3D reconstruction and then sends this to the content server. In a third further variant, the AR content server receives tracking information from an external tracking solution, and the server transforms it into the same coordinate system as used in the 3D reconstruction.

In a variant of an embodiment of the present application, the AR content server generates depth information of the light field once the light field data has been uploaded to the server. During runtime, when an AR content server receives a request for light field content, the content can be modified more efficiently (eliminating the need to perform depth detection during content delivery) by using depth information that has been previously generated.

Any of the implementations, variations, and permutations described in subsequent paragraphs and anywhere else in this disclosure can be implemented with respect to any implementation, including with respect to any method implementation and with respect to any system implementation.

The disclosed systems and methods can significantly reduce the amount of data transfer between the AR content service and the viewing client device in many online AR content delivery use cases. This reduction is based on the physical characteristics of the environment in which clients consume content, and can be performed on a per-client basis. This approach can be combined with other traditional data compression methods, thereby providing further content delivery optimization without sacrificing the benefits achievable with traditional content delivery compression techniques.

Description of drawings

In the drawings, the same reference numerals in each of the drawings represent the same or functionally similar elements, and the drawings are integrated in or form a part of the specification together with the following detailed description, and are used to further explain including claims Embodiments of the inventive concepts are presented and various principles and advantages of these embodiments are explained.

FIG. 1 is a visual overview of a method for AR content delivery in a pre-capture environment, according to at least one embodiment.

2 is a flowchart of a method performed by an AR client for delivery of AR content in a pre-capture environment, in accordance with at least one implementation.

3 is a flowchart of a method performed by an AR content server for delivery of AR content in a pre-capture environment, in accordance with at least one embodiment.

4 is a sequence diagram of a method for AR content delivery in a pre-capture environment, according to at least one embodiment.

5 is an example perspective view of a user in an example real-world AR viewing position, according to at least one implementation.

6 is a plan view of a digitally reconstructed 3D environment, according to at least one embodiment.

7 is a plan view of requested AR content and the digitally reconstructed 3D environment of FIG. 6, according to at least one embodiment.

FIG. 8 is a plan view exemplarily illustrating performing a visual analysis on the requested AR content of FIG. 7 according to at least one embodiment.

FIG. 9 is a plan view exemplarily showing modified AR content and the digitally reconstructed 3D environment of FIG. 6 according to at least one embodiment.

10 is an exemplary perspective view illustrating the modified AR content of FIG. 9 as seen by the user of FIG. 5 in accordance with at least one embodiment.

11 illustrates an example wireless transmit/receive unit (WTRU) that may be used as an AR client viewer in some implementations.

Figure 12 illustrates an example network entity that may act as an AR content server or may act as an AR content store in some implementations.

Skilled artisans appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the size of some of the elements in the figures may be exaggerated relative to other elements to help to improve understanding of embodiments of the invention.

Apparatus and method components are represented by conventional symbols where appropriate in the drawings, which merely show specific details that are relevant to an understanding of the embodiments of the invention so that the disclosure is not overwhelmed by details that are essential to an understanding of the description herein. Obvious to those skilled in the art) ambiguity.

Detailed ways

Before proceeding to the detailed description, it is noted that the entities, connections, arrangements, etc. depicted in and described in connection with the various figures are provided by way of example and not by way of limitation. Accordingly, any and all statements or other indications regarding the following (which may be taken in isolation and out of context to be absolute and thus limiting) may only be preceded constructively by words such as "In at least one embodiment ..." of what a particular figure "depicts", what a particular element or entity in a particular figure "is" or "has", and any and all Similar statements.

The present application discloses methods and systems for augmented reality content delivery in a pre-capture environment. This disclosure relates to content between an augmented reality (AR) content server and an AR client (e.g., an AR client running on an AR device, such as an optical-see-through AR headset or a video-see-through AR headset) data transmission. The exemplary method described herein utilizes a client device to provide a content server with information about the environment in which enhanced content is to be displayed. The AR content server adjusts AR content delivery based on the environmental characteristics of each individual client, whereby redundant data that is not visible to a particular client is removed from the content stream transmitted to the client.

In an exemplary embodiment, at the start of an AR session, the AR client device reconstructs a digital model of the current 3D environment. The generation of the model may use previously collected data corresponding to the same location. In other cases, the model is made by scanning the environment with sensors built into the AR client device, such as RGB-D or other depth sensors. A combination of previously collected data and currently collected data can be used to improve the accuracy of the model. A copy of the model is then sent to the content server, thereby giving the server instant access to the digital version of the current 3D environment. The server copy of the model can be updated and improved over time by incorporating more sensor data from sensing client devices. The model can be used to perform virtual object visibility analysis. The client device also uses the collected environmental data to assist in position and orientation tracking (attitude tracking). The pose of the device (viewpoint position and orientation) is used to synchronize the translation of augmented elements displayed by the AR client with the user's head movement. The pose information is sent to the server or calculated by the server using the environment model or other data to allow the server to estimate the visibility of elements of the AR content from the user's point of view. From the user's point of view, real world objects (eg, tables, buildings, trees, doors, and walls) can occlude AR content. This occurs when the depth of the AR content is further away from the user than real-world objects but are also on the same line of sight. Authentic AR content including high-resolution depth information allows for augmented reality perception.

In one embodiment, an AR device for viewing AR content is equipped with sensors capable of generating depth information from the environment. A sensor or combination of sensors may include one or more sensors such as RGB-D cameras, stereo cameras, infrared cameras, lidar, radar, sonar, and any other kind of sensor known to those skilled in the art of depth sensing. In some embodiments, a combination of sensor types and enhanced processing methods are used for depth detection. When a user initiates a viewing session, the viewing client reconstructs the environment model by collecting depth data from the environment.

During execution of the reconstruction process on the client device, the sensor collects point cloud data from the environment as the user moves the device and the sensor within the environment. Sensor observations with varying viewpoints are combined to form a coherent 3D reconstruction of the complete environment. Once the 3D reconstruction reaches the completion threshold, the AR client sends the reconstructed model and a request for specific AR content to the AR content server. In an exemplary embodiment, for example, the degree of completion of the 3D reconstruction may be measured in terms of percent coverage of the surrounding area, number of discrete observations, sensing duration, etc., and any similar quality value that can be used as a threshold. The 3D reconstruction of the environment can be performed using any known reconstruction method, such as those described in KinectFusion ^™ or the Point Cloud Library (PCL).

In one variation of this process, at the beginning of an AR viewing session, the client begins to continuously stream RGB-D data to the server. The AR client then sends a content request. In this variation, the AR content server performs a 3D reconstruction process using the received RGB-D data stream and stores the reconstructed environment model. As the server builds the per-client environment model, it also starts modifying the AR content by removing virtual elements occluded by the current environment. As the 3D reconstruction becomes more complete, the content removal process becomes more precise.

A user operating an AR client device may initiate a request for an AR session. This request is sent to the AR content server and indicates the specific content that the user has selected. The AR content to be displayed may be defined as a web link to the AR content server and a reference to specific AR content held within the AR content server. The operating system may automatically handle initializing the AR client based on the detection of the type of link activated by the user. The client may be embedded with an application for receiving the content link, such as a web browser and a mobile messaging application. Once the user starts an AR session, the previously described 3D reconstruction of the environment is initiated.

After the reconstruction process, the client starts the pose tracking process. The purpose of this pose tracking process is to estimate where the client device is and which direction the client device is facing relative to the previously reconstructed environment. Pose tracking can be performed using any known tracking technique and can be assisted using reconstructed environment models and client device sensor data. In some implementations, once the client device pose has been determined, the client sends the AR content server a reconstructed environment model and determined pose along with a request for specific AR content.

The AR content server receives the AR content request from the AR client, and obtains the requested content from a content store. The AR content server then modifies the requested AR content to remove redundant data. In this case, content delivery optimization involves a process performed by the AR content server. During content delivery optimization, the optimized portion of the requested AR content that is rendered at a location that is not visible to the viewer will be removed from the content. In AR, virtual elements of AR content are preferably perceived as (or merged with) the part of the real world that the viewer sees. In order to enhance reality perception, embodiments disclosed in the present application copy the appearance of physical objects of virtual objects that block the user's view to the virtual objects. By removing all elements from the content that are occluded by real-world elements, the size of the content can be reduced without a user-observable loss of quality. Depending on the type of content requested, the AR content server uses a slightly different content delivery optimization process. In fact, even client devices can take advantage of further improvements based on the type of AR content they request. Different approaches for optimizing light fields, photorealistic video, and compositing 3D scenes are described in the following sections.

Light field data can comprise an array of sub-images acquired simultaneously in a scene from several different viewpoints. Based on the array of sub-images, new viewpoints between original viewpoints can be generated computationally. The array of sub-images can also be processed to achieve refocusing of the compiled final image. Most relevantly, the array of sub-images may be used to extract depth information corresponding to the captured scene, eg via analysis of disparity and disparity maps of the sub-images.

For light field optimization, the AR content server extracts depth information from the light field and aligns the light scene depth map with the environment model received from the AR client. For each sub-image of the light field, the AR server transforms the viewpoint of the combined light scene depth map and the client environment model to match the viewpoint of the light field sub-image. For each pixel of the light field sub-image, the AR content server compares the depth value from the client's environment model with the corresponding depth value extracted from the light field data. If the depth map of the light field has a depth value greater than the depth value from the client's environment model, then the corresponding pixels in the light field sub-image are discarded because they are occluded by real world objects and thus are not visible to the client. visible.

As for photorealistic videos (which have associated depth information available), optimization can be performed by comparing the depth values of the photorealistic video with the depth values of the environment relative to the tracked camera pose. For any content removal, the AR content server operates to re-render the video. During re-rendering, portions of the video data that are occluded by real-world elements in the user's environment are discarded. These large dropout regions can be efficiently compressed using conventional video compression methods.

In some embodiments, a synthetic 3D scene including virtual 3D information may be optimized by performing visibility detection on virtual 3D elements within a geometry view captured from a real world viewing environment. In the visibility detection, the visibility of the content from the user's point of view is determined. From the geometry description of the virtual AR element, visibility detection can be performed with respect to objects or with respect to each vertex of each object. In some implementations, object-level removal (as opposed to vertex-level removal) is performed where the removed portion of the object's geometry results in the need for significant object data reorganization. This prevents 3D rendering algorithms from using buffer objects, which store parts of the content as static entities in GPU memory. As soon as the related content delivery optimization is completed, the modified AR content will be sent to the AR client.

The AR client receives the modified AR content from the AR content server and displays it to the viewer. To display the content, the AR client determines the device pose from the client device depth sensor. Based on the orientation information, the display process aligns the received content coordinate system with the user's coordinate system. In some implementations, after aligning the content, the sub-process compares the depth value received from the client device sensor to the depth value of the modified content received from the AR content server. Regions of the content with depth values greater than the corresponding depth values in the client device depth sensor can be discarded and not rendered because they would otherwise be rendered at spatial locations that are occluded by real-world physical elements in the environment. The AR content server already removes AR content elements that are occluded by elements present in the environment model sent to the server by the client, but this runtime depth comparison is able to handle missing dynamic elements and Occlusion caused by static elements.

Figure 1 is a visual overview of a method for AR content delivery in a pre-capture environment, according to at least one embodiment. A user 100 is depicted wearing an AR wearable client device 102 in the visual overview. The user 100 is in a real world AR viewing location 104 including a large table 106. The client device within 102 begins to reconstruct the current environment via the data collection cameras and sensors within the AR wearable client device 102. The client 102 collects data about the shape and position of the table using cameras and sensors in the AR wearable device and sends this data to the AR content server 108 .

The user 100 wishes to consume AR content depicting two virtual characters. The AR client sends a content request 114 to the AR content server 108 . The AR content server 108 performs visibility analysis on the selected content. The AR content server 108 determines that the top avatar is visible 110 and the bottom avatar 112 is occluded by the table 106 based on the user's position and orientation with respect to the current environment.

The AR content server 108 then modifies the AR content by removing the bottom avatar. Standard compression techniques can be used to reduce the size of the modified frame. The modified content stream is then sent from the server to the user 100 for consumption.

One embodiment includes a process performed by the AR wearable client device 102 . The process includes using the AR client device 102 to generate a digitally reconstructed 3D environment of the real world AR viewing location. The process consists of detecting objects in a digitally reconstructed 3D environment. The process also includes determining the depth and geometry of the object. The process also includes sending information indicative of the depth and geometry of the object to the AR content server 108 . The process also includes sending a request for AR content from the AR client to the AR content server 108 . The process also includes receiving, at the AR client device 102 , an AR content stream 116 from the AR content server 108 , wherein the received AR content stream 108 has been filtered by the AR content server 108 to exclude AR content that would be occluded by the object 112 .

2 is a flowchart of a method performed by an AR client for AR content delivery in a pre-capture environment, in accordance with at least one embodiment. FIG. 2 depicts process 200, which includes elements 202-210. At element 202, an AR client (eg, client 102) determines a real-world AR viewing location. At element 204 , the process 200 includes identifying and tracking a viewpoint position and orientation of an AR client (eg, client 102 ) within the real-world AR viewing location 104 . At element 206 , the process 200 includes sending information indicative of the viewpoint location and orientation to the AR content server 108 . At element 208 , the process 200 includes sending a request for AR content to the AR content server 108 . At element 201, the process 200 includes receiving, at the AR client device 102, an AR content stream, such as stream 116, from an AR content server, wherein the received AR content stream (e.g., stream 116) has been processed by the AR content server (e.g., , server 108) filters to exclude AR content that is not visible in the real-world AR viewing position from the viewpoint position and orientation.

3 is a flowchart of a method performed by an AR content server for delivery of AR content in a pre-capture environment, according to at least one embodiment. Figure 3 depicts a process 300 that includes elements 302-310. At element 302, an AR content server (eg, server 108) receives a request for AR content from an AR client (eg, client device 102). At element 304, the process 300 includes obtaining a digitally reconstructed 3D environment of a real world AR viewing location (eg, location 104). At element 306, the process 300 includes receiving a viewpoint location and orientation of the AR client for a real world AR viewing location. At element 308, the process 300 includes comparing the requested AR content to the digitally reconstructed 3D environment and viewpoint position and orientation to perform a visibility analysis of the requested AR content and The portion of the analysis indicated that is not visible from the viewpoint position and orientation is modified to modify the requested AR content. At element 310, the process 300 includes fulfilling the request for the AR content by sending the modified AR content to the AR client.

In at least one embodiment, obtaining a digitally reconstructed 3D environment of a real-world AR viewing location includes obtaining the reconstructed 3D environment from a data store. In at least one embodiment, obtaining the digitally reconstructed 3D environment of the AR viewing position in the real world includes obtaining the reconstructed 3D environment from the AR client. In at least one embodiment, obtaining a digitally reconstructed 3D environment of a real world AR viewing location includes generating the reconstructed 3D environment using data from a data store and real-time data from an AR client device.

In at least one embodiment of the AR content server, modifying the requested AR content varies by content type. In one such embodiment, the content type is a 3D scene, and modifying the requested AR content includes: (i) removing occluded 3D objects from the requested AR content, (ii) obtaining information about the viewpoint position and An update of the orientation, and (iii) based on the updated viewpoint position and orientation, the AR object that was removed earlier but is now visible is sent to the AR client. In another such embodiment, the content type is light field, and modifying the requested AR content includes: (i) processing the light field content frame by frame to remove occluded portions, and (ii) repackaging Remaining data for streaming. In another such embodiment, the content type is immersive video with depth data, and modifying the requested AR content includes: (i) removing the depth value ratio of the digitally reconstructed 3D environment in the video and The portion of the video whose associated depth value is smaller, (ii) re-renders the frame, and (iii) re-packages the video for streaming.

4 is a sequence diagram of a method for AR content delivery in a pre-capture environment, according to at least one embodiment. To serve as a further resource, the following paragraphs contain an example traversal of the sequence diagram of FIG. 4 .

The client device 102 reconstructs the current AR viewing position 404 in the reconstruction environment. The user 100 starts the AR viewing client 102 at step 406 and scans the current environment 408 using an AR headset that can be directly or wirelessly coupled to the viewing client. The client device 102 may alternatively determine the current AR viewing position and send this information to the AR content server 108 which already has a model of the environment. Scanning and reconstructing the environment is also useful for tracking purposes. In FIG. 4, when the digital reconstruction is complete, at step 410, the user 100 is notified that "client is ready".

The user 100 can then initiate an AR content streaming (viewing) session as shown in step 412 . The user 100 enters or selects the AR content link 414 and gives it to the AR client 102 . Upon receiving the link, the AR client performs pose tracking 414 (but this step could have been initiated at an earlier time). The AR client sends a content request 416 for the digitally reconstructed 3D environment and the current AR client pose to the AR content server 108 .

At step 420 , the AR content server 108 retrieves the requested AR content from the AR content store 402 at the fetch content 418 and at step 420 .

Content streaming and viewing 422 may begin after acquisition. The AR content server 108 optimizes the requested AR content 424 by removing elements occluded by the real world environment. This may involve modifying content frames, re-rendering modified frames, and so on. The AR content server 108 performs a visibility analysis (the AR client may also perform this) to determine which AR content is invisible to the user 100 using the digitally reconstructed 3D environment and client pose. The optimized content 424 will then be provided to the AR viewer client 102 via the optimized content stream 426 whereby the user 100 can see the displayed content 428 .

At least one embodiment includes a process for digitally reconstructing an augmented reality (AR) viewing position using an AR client. The process also includes sending information describing the digitally reconstructed AR viewing position from the AR client to the AR content server. The process also includes sending a request for AR content from the AR client to the AR content server. The process also includes sending information from the AR client to the AR content server describing the position and orientation of the AR client within the AR viewing location. The process also includes determining, at the AR content server, visibility of the requested AR content based on the received information describing the digitally reconstructed AR viewing position and the received information describing the position and orientation of the AR client. The process also includes modifying, at the AR content server, the requested AR content by removing portions of the requested AR content that are determined to be invisible. The process also includes sending the modified AR content from the AR content server to the AR client. The process also includes augmenting the AR viewing location with the modified AR content using the AR client.

5 is an illustration of a user 500 and an example real-world AR viewing location 502 in accordance with at least one implementation. FIG. 5 is a reference image used to assist the description of FIGS. 6 to 10 . FIG. 5 depicts a user 500 in an exemplary real-world AR viewing position. The viewing location 502 is a room that includes two doorways 504 and 506 , a dresser 508 , a sofa 510 , and a chair 512 . User 500 uses AR headset 514 to scan a real world AR viewing location.

6 is a plan view of a digitally reconstructed 3D environment, according to at least one embodiment. User 600 scans the AR viewing location using the AR client and generates a digitally reconstructed 3D environment. This digital reconstruction includes two doorways 602 and 604 , a dresser 606 , a sofa 608 and a chair 610 . Figure 2 is a plan view of the reconstruction. The AR headset worn by the user 600 determines its pose and sends the pose and the digital reconstruction to the AR content server 108 .

7 is an illustration of requested AR content and the digitally reconstructed 3D environment of FIG. 6, according to at least one embodiment. User 600 has requested AR content depicting nine stars 700 (eg, in an AR planetarium application). The coordinate system of the AR content is aligned with the real world viewing position by using digitally reconstructed 3D environment and pose information. This alignment can be performed in a number of ways. For example, the alignment may be chosen to minimize the amount of AR content (such as the number of stars) that would not be displayed due to occlusion. The alignment may be selected using user input, for example the user may adjust the alignment until a preferred alignment is selected. The alignment may be selected based on other inputs, for example one or more accelerometers in the AR display device may be used to ensure that the "up" direction with respect to the AR content is aligned with the "up" direction of the AR viewing environment. In some implementations, the alignment can be selected such that the direction the user is facing at the start of the AR session can be aligned with the preferred viewing direction of the AR content.

Based on the selected alignment, the display position of each element of the AR content (such as the display position of each star) can be transformed from the coordinate system of the AR content to the coordinates of the 3D viewing environment by using the selected alignment system to determine.

8 is an illustration of a visibility analysis performed on the requested AR content of FIG. 7 in accordance with at least one embodiment. The AR content server 108 performs visibility analysis using the digitally reconstructed 3D environment and pose information. The AR content server 108 determines that the four stars 802, 804, 806, and 808 included in the original AR content are not visible from the viewpoint of the user 600, and removes them from the content.

9 is an illustration of modified AR content and the digitally reconstructed 3D environment of FIG. 6 in accordance with at least one embodiment. The user has requested AR content depicting nine stars, but the AR content server has modified the AR content and now only five stars appear in the modified data stream.

10 is an illustration of the modified AR content of FIG. 9 as seen by the user of FIG. 5 in accordance with at least one embodiment. Figure 10 shows a real world environment augmented by modified AR content. The 5 remaining stars 1002, 1004, 1006, 1008 and 1010 are all visible. The 4 removed stars 802, 804, 806 and 808 are not obscured by walls, chairs, etc., but none of them appear in the data stream.

The exemplary embodiments disclosed herein are implemented using one or more wired and/or wireless network nodes, such as wireless transmit/receive units (WTRUs) or other network entities.

Figure 11 is a system diagram of an exemplary WTRU 1102 that may function as an AR client viewer in embodiments of the present application. As shown in FIG. 11 , WTRU 1102 may include processor 1118, communication interface 1119 including transceiver 1120, transmit/receive component 1122, speaker/microphone 1124, keyboard 1126, display/touchpad 1128, non-removable memory 1130 , removable memory 1132 , power supply 1134 , global positioning system (GPS) chipset 1136 , and sensor 1138 . It should be appreciated that the WTRU 1102 may include any subcombination of the foregoing components while remaining consistent with the embodiments.

Processor 1118 can be a general purpose processor, a special purpose processor, a conventional processor, a digital signal processor (DSP), multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller , application-specific integrated circuits (ASICs), field-programmable gate array (FPGA) circuits, any other type of integrated circuit (IC), and state machines, among others. Processor 1118 may perform signal encoding, data processing, power control, input/output processing, and/or any other functions that enable WTRU 1102 to operate in a wireless environment. Processor 1118 can be coupled to transceiver 1120 , which can be coupled to transmit/receive component 1122 . Although FIG. 11 depicts the processor 1118 and the transceiver 1120 as separate components, it should be understood that the processor 1118 and the transceiver 1120 may also be integrated in one electronic component or chip.

Transmit/receive component 1122 may be configured to transmit or receive signals to or from a base station via air interface 1116 . For example, in one embodiment, transmit/receive component 1122 may be an antenna configured to transmit and/or receive RF signals. As an example, in another embodiment, the transmit/receive component 1122 may be an emitter/detector configured to emit and/or receive IR, UV, or visible light signals. In yet another embodiment, the transmit/receive component 1122 may be configured to transmit and receive RF and optical signals. It should be appreciated that the transmit/receive component 1122 may be configured to transmit and/or receive any combination of wireless signals.

Furthermore, although the transmit/receive component 1122 is depicted in FIG. 11 as a single component, the WTRU 1102 may include any number of transmit/receive components 1122 . More specifically, the WTRU 1102 may use MIMO technology. Thus, in one embodiment, the WTRU 1102 may include two or more transmit/receive components 1122 (eg, multiple antennas) for transmitting and receiving radio signals over the air interface 1116 .

The transceiver 1120 may be configured to modulate signals to be transmitted by the transmit/receive component 1122 and to demodulate signals received by the transmit/receive component 1122 . As noted above, the WTRU 1102 may be multimode capable. Accordingly, the transceiver 1120 may include multiple transceivers that allow the WTRU 1102 to communicate over multiple RATs (eg, UTRA and IEEE 802.11).

Processor 1118 of WTRU 1102 may be coupled to speaker/microphone 1124, keyboard 1126, and/or display/touchpad 1128 (such as a liquid crystal display (LCD) display unit or an organic light emitting diode (OLED) display unit) and may receive user input data. Processor 1118 may also output user data to speaker/microphone 1124 , keyboard 1126 and/or display/touchpad 1128 . In addition, processor 1118 may access information from, and store information in, any suitable memory, such as non-removable memory 1130 and/or removable memory 1132 . The non-removable memory 1130 may include random access memory (RAM), read only memory (ROM), hard disk, or any other type of memory storage device. Removable memory 1132 may include a Subscriber Identity Module (SIM) card, a memory stick, a Secure Digital (SD) memory card, and the like. In other embodiments, the processor 1118 may access information from, and store data into, memory that is not physically located on the WTRU 1102, such memory may be located on a server or home computer (not shown), as examples.

Processor 1118 may receive power from power supply 1134 and may be configured to distribute and/or control power for other components in WTRU 1102 . Power source 1134 may be any suitable device for powering WTRU 1102 . For example, the power source 1134 may include one or more dry battery packs (such as nickel cadmium (Ni-Cd), nickel zinc (Ni-Zn), nickel metal hydride (NiMH), lithium ion (Li-ion), etc.), solar cells, and fuel cells and more.

The processor 1118 may also be coupled to a GPS chipset 1136 that may be configured to provide location information (eg, longitude and latitude) related to the current location of the WTRU 1102 . In addition to or instead of information from GPS chipset 1136, WTRU 1102 may receive location information from base stations via air interface 1116 and/or determine its location based on signal timing received from two or more nearby base stations. It should be appreciated that the WTRU 1102 may obtain location information by any suitable positioning method while remaining consistent with the embodiments.

The processor 1118 may also be coupled to other peripheral devices 1138, which may include one or more software and/or hardware modules that provide additional features, functionality, and/or wired or wireless connectivity. For example, peripheral devices 1138 may include accelerometers, electronic compasses, satellite transceivers, digital cameras (for photo or video), Universal Serial Bus (USB) ports, vibrating devices, television transceivers, hands-free headsets, modules, frequency modulation (FM) radio units, digital music players, media players, video game console modules, Internet browsers, and more.

FIG. 12 shows an exemplary network entity 1290 that may be used in embodiments of the present disclosure, eg, as an AR content server or as an AR content store. As shown in FIG. 12 , network entity 1290 includes communication interface 1292 , processor 1294 , and non-transitory data storage 1296 , all of which are communicatively linked by bus, network, or other communication path 1928 .

Communication interface 1292 may include one or more wired communication interfaces and/or one or more wireless communication interfaces. For wired communications, communication interface 1292 may include one or more interfaces, such as an Ethernet interface. With respect to wireless communications, the communications interface 1292 may include a number of components, such as one or more antennas, one or more transceivers/chipsets designed and configured for one or more types of wireless communications (e.g., LTE), and/or any other components considered appropriate by those skilled in the art. Further, regarding wireless communication, the communication interface 1292 may be sized with a configuration suitable for use as wireless communication (eg, LTE communication, Wi-Fi communication, etc.) on the network side (as opposed to the client side). Accordingly, communication interface 1292 may comprise suitable equipment and circuitry (possibly including multiple transceivers) for serving multiple mobile stations, UEs or other access terminals in the coverage area.

Processor 1294 may include one or more processors of any type deemed suitable by those skilled in the art, some examples include general purpose microprocessors and special purpose DSPs.

Data storage 1296 may take the form of any non-transitory computer-readable medium or combination of these, some examples include flash memory, read-only memory (ROM), and random-access memory (RAM), as recognized by those skilled in the art. Any one or more types of non-transitory data storage used. As shown in FIG. 12, data store 1296 contains program instructions 1297 executable by processor 1294 for performing various combinations of various network entity functions described herein.

In the foregoing description, specific embodiments have been described. However, it is understood by those skilled in the art that various modifications and changes can be made without departing from the scope of the present invention as set forth in the claims. Accordingly, the specification and figures are to be regarded as illustrative rather than restrictive, and all such modifications are included within the teaching of the invention.

The existence or becoming apparent of benefits, advantages, solutions to problems and any element that would bring about any benefit, advantage or solution is not to be construed as a critical, required or essential feature or element of any or all of the claims. The invention is defined solely by the appended claims including any amendments made during the pendency of this application and all equivalents of such claims as issued.

Also in this document, relative terms (e.g., first and second, top and bottom, etc.) may be used solely to distinguish one entity or action from another without necessarily requiring or implying a relationship between these entities or actions. any actual relationship or sequence between them. The terms "comprises", "comprising", "has", "having", "includes", "including", "contains", " "containing" or any other variation thereof is intended to include open inclusion, whereby a process, method, article, or apparatus that includes, has, comprises, or contains listed elements includes not only those elements, but may also include elements not expressly listed or other elements inherent in these processes, methods, articles, or apparatus. An element that is "comprised," "has," "comprises," or "contains" does not, without further limitation, exclude the inclusion, The process, method, article or apparatus of the same element is otherwise present. The terms "a" and "an" are defined as one or more, unless expressly stated otherwise in this application. The term " Substantially", "essentially", "approximately", "approximately" or any other version thereof are defined as close to what a person skilled in the art would understand, and in one non-limiting embodiment the term is defined as being within 10% of , within 5% in another embodiment, within 1% in another embodiment, and within 0.5% in another embodiment. The term "coupled" as used in this application is defined as connecting, but not Must be a direct connection and not necessarily a mechanical connection. A device or structure that is "configured" in a certain way is configured in at least that way, but may also be configured in ways that are not listed.

It is understood that some embodiments include one or more general or special purpose processors (or "processing devices"), such as microprocessors, digital signal processors, custom processors, and field programmable gate arrays (FPGAs), as well as uniquely stored Program instructions (including software and firmware) that control one or more processors in conjunction with certain non-processor circuits to perform some, most or all of the functions of the methods and/or apparatuses described herein. Alternatively, some or all functions may be implemented by a state machine with no stored program instructions, or in one or more application-specific integrated circuits (ASCIs), where each function or some combination of certain functions are implemented as custom logic . Of course, a combination of both approaches can be used.

Accordingly, some embodiments of the present disclosure, or portions thereof, may combine one or more processing devices and one or more software components (e.g., program code, firmware, resident software, microcode) stored in a tangible computer-readable storage device etc.), which combinations form a specifically configured apparatus for performing the functions described herein. These combinations forming a specially programmed device may collectively be referred to as "modules" in this application. The software component parts of the module may be written in any computer language and may be part of a monolithic code library, or may be developed in more discrete code parts (such as typically in an object-oriented computer language). Furthermore, the modules may be distributed to multiple computer platforms, servers, terminals, etc. A given module may even be implemented as a separate processor device and/or computing hardware platform to perform the described functions.

Furthermore, the embodiments can be implemented as a computer-readable storage medium having stored thereon computer-readable code for programming a computer (eg, comprising a processor) to perform the methods described and claimed herein. Examples of computer-readable storage media include, but are not limited to, hard drives, CD-ROMs, optical storage devices, magnetic storage devices, ROM (Read Only Memory), PROM (Programmable Read Only Memory), EPROM (Erasable Programmable Read Only Memory), Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), and Flash memory. Furthermore, it is envisioned that those skilled in the art, while motivated, for example, by available time, current technology, and economic considerations, will make potentially significant effort and many design choices, who, guided by the concepts and principles disclosed in the present application, will make it with minimal experimentation. It is easy to be able to generate these software instructions and programs as well as the IC in the case.

The Abstract of the Disclosure is provided to allow the reader to quickly ascertain the nature of the technical disclosure. It is to be understood that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description section, it can be seen that various features are combined in various implementations in order to streamline the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, the claims reflect inventive subject matter in less than all features of a single disclosed embodiment. Thus the claims are hereby incorporated into the Detailed Description, with each claim standing on its own as a separately claimed subject matter.

Claims (16)

1. a kind of method, comprising:

Obtain three-dimensional (3D) environment of digital reconstruction of augmented reality (AR) viewing location；

Detect the object in the digital reconstruction 3D environment；

Determine the depth and geometry of the object；

The request for being directed to augmented reality (AR) content is sent to server, the request includes indicating the depth of the object With the information of the geometry；

The AR content stream is received from the server, the AR content stream excludes the AR content blocked by the object.

2. according to the method described in claim 1, further including using global positioning system (GPS) data, wireless data and figure The AR viewing location is determined as one or more of identification.

3. according to the method described in claim 1, wherein obtaining the digital reconstruction 3D via one or more of following Environment:

AR session before storage；

The combination of the data of the data and real-time collecting of storage；Or

Depth data point cloud.

4. according to the method described in claim 1, wherein via by checking the AR viewing location to described from multiple viewpoints AR viewing location is scanned to obtain the digital reconstruction 3D environment.

5. according to the method described in claim 1, further include:

Track the viewpoint position and direction in the digital reconstruction 3D environment.

6. according to the method described in claim 1, wherein indicate the object the depth and the geometry it is described Information includes one or more of the following:

Real-time original sensor data；

The depth map of the object；Or

RGB-D data flow.

7. according to the method described in claim 1, wherein the type of requested AR content be intend true video, and instruction described in The depth of at least one real world objects and the information of the geometry are formatted as the real world The spherical depth map of AR viewing location.

8. a kind of augmented reality (AR) content server, comprising:

Processor；

Communication interface；And

Non-transient data storage；

The AR content server is configured to:

The request for being directed to AR content is received via the communication interface；

Requested AR content is obtained from data storage；

Obtain three-dimensional (3D) environment of digital reconstruction of AR viewing location；

Viewpoint position and direction in AR viewing location are received via the communication interface；

It is executed compared with the digital reconstruction 3D environment and the viewpoint position and direction via requested AR content Visualization analysis；

By according to the visualization analysis remove with from the viewpoint position and towards sightless one or more object phases Associated data modify requested AR content.

9. AR content server according to claim 8, wherein from one of following or a variety of acquisitions number weight Structure 3D environment:

Data storage；

Augmented reality (AR client)；Or

The combination of the data storage and the AR client.

10. AR content server according to claim 8, wherein the modification of requested AR content is according to being requested AR content content type, the content type includes one or more of the following:

Three-dimensional (3D) scene；

Light field；Or

Intend true video with depth data.

11. AR content server according to claim 8, wherein the requested AR content of the modification includes:

Viewpoint position and removed three-dimensional (3D) object towards before being inserted into requested AR content based on update.

12. AR content server according to claim 10, wherein the visualization analysis includes:

Identify that the depth value in which place in the digital reconstruction 3D environment is less than depth associated with requested AR content Data.

13. a kind of method, comprising:

Receive the request for being directed to augmented reality (AR) audio content stream；

Requested AR audio content stream is obtained from data storage；

Obtain the digital reconstruction 3D environment of real world AR viewing location；

Receive the viewpoint position and direction of the AR client of the real world AR viewing location；

Analyze the digital reconstruction 3D environment and the viewpoint position and towards with the requested AR audio content stream of determination whether Including the one or more objects being blocked in the digital reconstruction 3D environment；And

Remove the one or more of objects being blocked in the digital reconstruction 3D environment.

14. according to the method for claim 13, further includes:

Show that equipment sends the AR audio content stream of modification to AR.

15. according to the method for claim 14, wherein the analysis digital reconstruction 3D environment and the viewpoint position And direction includes:

Determine that the AR shows the position of equipment；

It is identified one in the digital reconstruction 3D environment along the viewpoint position for showing equipment from the AR and towards determining sight A or multiple physical objects；And

Determine one or more of objects in the AR audio content stream whether by one or more based on above-mentioned sight A physical object blocks.

16. according to the method for claim 14, wherein the analysis digital reconstruction 3D environment and the viewpoint position Include: with direction

The depth value of the digital reconstruction 3D environment is compared with the depth value in the AR audio content stream；And

The part that depth value in the AR audio content stream is greater than the correspondence depth value in the digital reconstruction 3D environment is abandoned, with It compensates and is blocked caused by the one or more dynamic objects and/or static object lacked from the digital reconstruction 3D environment.