CN118567475A - Apparatus, method and graphical user interface for interacting with a three-dimensional environment - Google Patents

Abstract

The present disclosure relates to devices, methods, and graphical user interfaces for interacting with a three-dimensional environment. A computer system simultaneously displays: a view of the physical environment; and a computer-generated user interface element overlaid on the view of the physical environment. The appearance of the computer-generated user interface element is based on the appearance of the view of the physical environment covered by the computer-generated user interface element. In response to a change in appearance of the physical environment, the appearance of the computer-generated user interface element is updated at a first time based on a graphical composition of the appearance at a different time before the first time of one or more portions of the physical environment, comprising: an appearance of a first portion of the physical environment at a second time prior to the first time; and an appearance of a second portion of the physical environment at a third time prior to the second time.

Description

Apparatus, method and graphical user interface for interacting with a three-dimensional environment

The present application is a divisional application of the application patent application with international application number 2022, 9, 23, national application number 202280064827.1 (international application number PCT/US 2022/044577), entitled "apparatus, method and graphical user interface for interacting with three-dimensional environments".

Related patent application

The present application is a continuation of U.S. patent application Ser. No. 17/950,035, issued to 2022, 9 and 21, claiming priority from U.S. provisional application Ser. No. 63/328,686, issued to 2022, 4 and 7, and U.S. provisional application Ser. No. 63/248,381, issued to 2021, 9 and 24, the entire contents of each of which are incorporated herein by reference in their entirety.

Technical Field

The present disclosure relates generally to computer systems having a display generating component and one or more input devices that provide a computer-generated augmented reality (XR) experience, including but not limited to electronic devices that provide virtual reality and mixed reality experiences via the display generating component.

Background

In recent years, the development of computer systems for virtual reality, augmented reality, and augmented reality has increased significantly. Exemplary augmented reality and augmented reality environments include at least some virtual elements that replace or augment the physical world. Input devices (such as cameras, controllers, joysticks, touch-sensitive surfaces, and touch screen displays) for computer systems and other electronic computing devices are used to interact with the virtual/augmented reality environment. Exemplary virtual elements include virtual objects such as digital images, videos, text, icons, and control elements such as buttons and other graphics.

Methods and interfaces for interacting with environments that include at least some virtual elements (e.g., applications, augmented reality environments, mixed reality environments, virtual reality environments, and augmented reality environments) are cumbersome, inefficient, and limited. For example, providing a system for insufficient feedback of actions associated with virtual objects, a system that requires a series of inputs to achieve desired results in a virtual/augmented reality environment, and a system in which virtual objects are complex, cumbersome, and error-prone to manipulate, can create a significant cognitive burden on the user and detract from the feel of the virtual/augmented reality environment. In addition, these methods take longer than necessary, wasting energy. This latter consideration is particularly important in battery-powered devices.

Disclosure of Invention

Accordingly, there is a need for a computer system with improved methods and interfaces to provide a user with a computer-generated experience, thereby making user interactions with the computer system more efficient and intuitive for the user. The disclosed systems, methods, and user interfaces reduce or eliminate the above-described drawbacks and other problems associated with user interfaces for computer systems having a display generating component and one or more input devices. Such systems, methods, and interfaces optionally supplement or replace conventional systems, methods, and user interfaces for providing an augmented reality experience to a user. Such methods and interfaces reduce the number, extent, and/or nature of inputs from a user by helping the user understand the association between the inputs provided and the response of the device to those inputs, thereby forming a more efficient human-machine interface.

According to some embodiments, a method is performed at a computer system in communication with a display generation component. The method includes simultaneously displaying, via a display generating component, a view of a physical environment and a computer-generated user interface element overlaid on at least a portion of the view of the physical environment. The appearance of the computer-generated user interface element is based on an appearance of one or more portions of a view of a physical environment covered by the computer-generated user interface element. When a view of a physical environment and a computer-generated user interface element are displayed simultaneously, the appearance of a portion of the physical environment changes. In response to a change in appearance of the portion of the physical environment, the appearance of the computer-generated user interface element is updated at a first time based on a graphical composition of the appearance at a different time before the first time of the one or more portions of the physical environment, the appearance at a different time before the first time of the one or more portions of the physical environment including: an appearance of a first portion of the physical environment at a second time prior to the first time; and an appearance of a second portion of the physical environment at a third time prior to the second time.

In some embodiments, a method is performed at a computer system in communication with a display generation component. The method includes simultaneously displaying, via a display generating component: a view of the physical environment; and a computer-generated user interface element overlaid on at least a portion of the view of the physical environment. The appearance of the computer-generated user interface element is based on a respective gradient between a first set of representative colors sampled from one or more portions of a view of a physical environment covered by the computer-generated user interface element. The first set of representative colors includes: a first color representing a portion of a view of the physical environment behind a first portion of the computer-generated user interface element over a first period of time; and a second color representing a portion of the view of the physical environment behind the second portion of the computer-generated user interface element within the first period of time. When a view of a physical environment and a computer-generated user interface element are displayed simultaneously, the appearance of a portion of the physical environment changes. The method comprises the following steps: in response to a change in appearance of the portion of the physical environment, an appearance of the computer-generated user interface element is updated based on a respective gradient between a second set of representative colors sampled from one or more portions of the view of the physical environment covered by the computer-generated user interface element. The second set of representative colors includes: a third color representing a portion of the view of the physical environment behind the first portion of the computer-generated user interface element within a second time period different from the first time period; and a fourth color representing a portion of the view of the physical environment behind the second portion of the computer-generated user interface element over a second period of time.

In some embodiments, a method is performed at a computer system in communication with a display generation component. The method includes displaying, via a display generating component, a first view of the three-dimensional environment, including displaying a first user interface object in the first view of the three-dimensional environment and displaying background content behind the first user interface object in the three-dimensional environment. The first user interface object includes a first portion having an appearance selected based on an appearance of background content located behind the first portion of the first user interface object in the three-dimensional environment. The first user interface object has a simulated thickness between the first portion and a front of the first user interface object; and the content is displayed between a first portion of the first user interface object and a front portion of the first user interface object. The method comprises the following steps: detecting a request for mobile content while the first user interface object is displayed; and in response to detecting the request to move the content, moving the content within the first user interface object while the content remains between the first portion of the first user interface object and the front of the first user interface object. Moving the content within the first user interface object includes displaying a visual effect applied to the content based on the simulated thickness of the first user interface object.

In some embodiments, a method is performed at a computer system having a display generating component and one or more input devices. The method comprises the following steps: display via a display generating means: a user interface comprising a view of a three-dimensional environment; a simulated three-dimensional object in a three-dimensional environment, the simulated three-dimensional object being displayed with an appearance corresponding to a respective simulated material, wherein a first region of the simulated three-dimensional object is displayed with an appearance that removes a portion of the respective simulated material from a surface of the simulated three-dimensional object; and a first user interface element comprising a first region of the simulated three-dimensional object. The method comprises the following steps: detecting a first input directed to a first user interface element; and in response to detecting the first input directed to the first user interface element: a respective operation associated with the first user interface element is performed in the user interface.

In some embodiments, a method is performed at a computer system having a display generating component and one or more input devices. The method includes displaying, via a display generation component, a user interface including a view of a three-dimensional environment and user interface elements associated with respective surfaces in the three-dimensional environment. A first set of one or more portions of the user interface element is associated with a first layer of the user interface element; a second set of one or more portions of the user interface element is associated with a second layer of the user interface element; and the user interface element is displayed having an appearance indicating a first degree of separation between the first layer and the second layer. The method comprises the following steps: detecting a first input corresponding to a user focusing attention on a user interface element; and in response to detecting a first input corresponding to the user focusing attention on the user interface element: the appearance of the user interface element is changed to indicate a second degree of separation between the first layer and the second layer. The second degree is different from the first degree.

In some embodiments, a method is performed at a computer system in communication with a display generation component and one or more input devices. The method includes displaying, via a display generating component, a user interface object while a first view of the three-dimensional environment is visible. The user interface object includes a first selectable object that can be activated to perform a corresponding operation in a three-dimensional environment. The method includes detecting a first input corresponding to a first selectable object. The method further comprises the steps of: in response to detecting the first input corresponding to the first selectable object, and in accordance with a determination that the first input indicates that the user is ready to interact with the first selectable object, a thickness of the first selectable object is changed.

In some embodiments, a method is performed at a computer system having a display generating component and one or more input devices. The method includes displaying, via a display generating component, a computer-generated user interface object overlaid on a first portion of a view of the three-dimensional environment. The appearance of the computer-generated user interface object is based on the blurred representation of the first portion of the view of the three-dimensional environment covered by the computer-generated user interface object. The method includes visually distinguishing a first region of a computer-generated user interface object from a second region of the computer-generated user interface object by: applying a first transformation to a first subset of the blurred representation of the first portion of the view of the three-dimensional environment covered by the computer-generated user interface object, the first subset corresponding to a first region of the computer-generated user interface object, without applying a second transformation; and applying a second transformation to a second subset of the blurred representation of the first portion of the view of the three-dimensional environment covered by the computer-generated user interface object, the second subset corresponding to a second region of the computer-generated user interface object, without applying the first transformation. The second subset of blurred representations of the first portion of the view of the three-dimensional environment is different from the first subset of blurred representations of the first portion of the view of the three-dimensional environment. The method comprises the following steps: as the appearance of the first portion of the view of the three-dimensional environment changes, updating the appearance of the computer-generated user interface object based on the blurred representation of the changed first portion of the view of the three-dimensional environment, comprising: applying a first transformation to a first subset of the blurred representation of the changed first portion of the view of the three-dimensional environment covered by the computer-generated user interface object, the first subset corresponding to a first region of the computer-generated user interface object, without applying a second transformation; and applying a second transformation to a second subset of the blurred representation of the changed first portion of the view of the three-dimensional environment covered by the computer-generated user interface object, the second subset corresponding to a second region of the computer-generated user interface object, without applying the first transformation. The second subset of blurred representations of the changed first portion of the view of the three-dimensional environment is different from the first subset of blurred representations of the changed first portion of the view of the three-dimensional environment.

According to some embodiments, the computer system includes or communicates with: a display generation component (e.g., a hardware element comprising one or more display devices such as a display, projector, heads-up display, head-mounted display, touch screen, etc.), one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors for detecting contact strength with the touch-sensitive surface), optionally one or more audio output components, optionally one or more haptic output generators, one or more processors, and a memory storing one or more programs; the one or more programs are configured to be executed by the one or more processors, and the one or more programs include instructions for performing or causing the operations of performing any of the methods described herein. According to some embodiments, a non-transitory computer-readable storage medium has stored therein instructions that, when executed by a computer system having a display generating component, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors for detecting contact strength with the touch-sensitive surface), optionally one or more audio output components, and optionally one or more tactile output generators, cause the device to perform any of the methods described herein or cause the operations of any of the methods described herein to be performed. According to some embodiments, a graphical user interface on a computer system having a display generating component, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors for detecting intensity of contact with the touch-sensitive surface), optionally one or more audio output components, optionally one or more haptic output generators, a memory, and one or more processors for executing one or more programs stored in the memory, includes one or more elements of the elements displayed in any of the methods described herein, the one or more elements updated in response to an input, as described in any of the methods described herein. According to some embodiments, a computer system includes: a display generating component, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors for detecting intensity of contact with the touch-sensitive surface), optionally one or more audio output components, and optionally one or more tactile output generators; and means for performing or causing the operations of any one of the methods described herein. According to some embodiments, an information processing apparatus for use in a computer system having a display generating component, one or more input devices (e.g., one or more cameras, a touch-sensitive surface, optionally one or more sensors for detecting intensity of contact with the touch-sensitive surface), optionally one or more audio output components, and optionally one or more tactile output generators, comprises means for performing or causing to be performed the operations of any of the methods described herein.

Accordingly, improved methods and interfaces are provided for computer systems having display generating components for interacting with a three-dimensional environment and facilitating user use of the computer system in interacting with the three-dimensional environment, thereby improving the effectiveness, efficiency, and user safety and satisfaction of such computer systems. Such methods and interfaces may supplement or replace conventional methods for interacting with a three-dimensional environment and facilitating user use of a computer system in interacting with the three-dimensional environment.

It is noted that the various embodiments described above may be combined with any of the other embodiments described herein. The features and advantages described in this specification are not all-inclusive and, in particular, many additional features and advantages will be apparent to one of ordinary skill in the art in view of the drawings, specification, and claims. Furthermore, it should be noted that the language used in the specification has been principally selected for readability and instructional purposes, and may not have been selected to delineate or circumscribe the inventive subject matter.

Drawings

For a better understanding of the various described embodiments, reference should be made to the following detailed description taken in conjunction with the following drawings, in which like reference numerals designate corresponding parts throughout the several views.

FIG. 1 is a block diagram illustrating an operating environment of a computer system for providing an augmented reality (XR) experience, according to some embodiments.

FIG. 2 is a block diagram illustrating a controller of a computer system configured to manage and coordinate a user's XR experience, according to some embodiments.

FIG. 3 is a block diagram illustrating a display generation component of a computer system configured to provide a visual component of an XR experience to a user, according to some embodiments.

FIG. 4 is a block diagram illustrating a hand tracking unit of a computer system configured to capture gesture inputs of a user, according to some embodiments.

Fig. 5 is a block diagram illustrating an eye tracking unit of a computer system configured to capture gaze input of a user, according to some embodiments.

Fig. 6 is a flow diagram illustrating a flash-assisted gaze tracking pipeline in accordance with some embodiments.

Fig. 7A-7G illustrate displaying a user interface element on a view of a physical environment, where the user interface element has an appearance based on a graphical composition of content in the physical environment behind the user interface element at a plurality of different points in time, according to some embodiments.

Fig. 7H-7J illustrate displaying a user interface element on a view of a three-dimensional environment, where the user interface element has an appearance based on a color gradient generated from one or more colors sampled from a plurality of points in the three-dimensional environment, according to some embodiments.

Fig. 7K-7O illustrate displaying a user interface element having the appearance of a simulated material with simulated thickness and curvature along an edge of the user interface element, where the edge of the user interface element has simulated optical properties that interact with light in a three-dimensional environment, according to some embodiments.

FIG. 7P illustrates separating layers of user interface elements to different extents in response to different user interactions including a user focusing attention on the user interface elements, in accordance with some embodiments.

Fig. 7Q-7W illustrate changing an appearance of a selectable user interface object in response to an input corresponding to the selectable user interface object and indicating that interaction with the selectable user interface object is ready, according to some embodiments.

Fig. 7X-7Z illustrate visually distinguishing different portions of a computer-generated user interface object whose appearance is based on content in a surrounding three-dimensional environment behind the user interface object by applying different transformations to representations of underlying content, according to some embodiments.

FIG. 8 is a flow chart of a method of displaying a user interface element on a view of a physical environment, where the user interface element has an appearance based on a graphical composition of content in the physical environment behind the user interface element at a plurality of different points in time, according to some embodiments.

FIG. 9 is a flow chart of a method of displaying a user interface element on a view of a three-dimensional environment, where the user interface element has an appearance based on a color gradient generated from one or more colors sampled from a plurality of points in the three-dimensional environment, according to some embodiments.

FIG. 10 is a flowchart of a method of displaying a user interface element having the appearance of simulated material with simulated thickness and curvature along an edge of the user interface element, where the edge of the user interface element has simulated optical properties that interact with light in a three-dimensional environment, according to some embodiments.

FIG. 11 is a flowchart of a method of displaying a user interface element having the appearance of simulated material with simulated thickness and curvature along an edge of the user interface element, where the edge of the user interface element has simulated optical properties that interact with light in a three-dimensional environment, according to some embodiments.

FIG. 12 is a flow chart of a method of separating layers of a user interface element to different extents in response to different user interactions including a user focusing attention to the user interface element, according to some embodiments.

FIG. 13 is a flow chart of a method of changing the appearance of a selectable user interface object in response to an input corresponding to the selectable user interface object and indicating that interaction with the selectable user interface object is ready, according to some embodiments.

FIG. 14 is a flowchart of a method of visually distinguishing different portions of a computer-generated user interface object whose appearance is based on content in a surrounding three-dimensional environment behind the user interface object by applying different transformations to a representation of underlying content, according to some embodiments.

Detailed Description

According to some embodiments, the present disclosure relates to a user interface for providing a computer-generated augmented reality (XR) experience to a user.

The systems, methods, and GUIs described herein improve user interface interactions with virtual/augmented reality environments in a variety of ways.

In some embodiments, the computer system simultaneously displays a view of the physical environment and a computer-generated user interface element (e.g., an application user interface, a view of a file or document, or a communication session) overlaid on at least a portion of the view of the physical environment. In some embodiments, the appearance of the computer-generated user interface element is based on the appearance of one or more portions of the view of the physical environment covered by the computer-generated user interface element. In some embodiments, the appearance of the computer-generated user interface element is based on a graphical composition of the appearance of one or more portions of the physical environment at different times. In some embodiments, in response to a change in appearance of the portion of the physical environment covered by the computer-generated user interface element, the appearance of the computer-generated user interface element is updated at the first time based on a graphical composition of appearances of one or more portions of the physical environment at different times prior to the first time, the appearances of the one or more portions of the physical environment at different times prior to the first time including an appearance of the first portion of the physical environment at a second time prior to the first time; and an appearance of a second portion of the physical environment at a third time prior to the second time.

In some embodiments, the computer system simultaneously displays a view of the physical environment and a computer-generated user interface element (e.g., an application user interface, a view of a file or document, or a communication session) overlaid on at least a portion of the view of the physical environment. In some embodiments, the appearance of the computer-generated user interface element is based on a respective gradient between a first set of representative colors sampled from one or more portions of the view of the physical environment covered by the computer-generated user interface element. In some embodiments, the first set of representative colors includes at least two colors sampled from different portions of the view of the physical environment that are behind different portions of the computer-generated user interface element over the first period of time. In some embodiments, in response to a change in appearance of the portion of the physical environment covered by the computer-generated user interface element, the computer system updates the appearance of the computer-generated user interface element based on a respective gradient between a second set of representative colors sampled from one or more portions of the view of the physical environment covered by the computer-generated user interface element. In some embodiments, the second set of representative colors includes at least two colors sampled from different portions of the view of the physical environment that are behind different portions of the computer-generated user interface element within a second period of time that is different from the first period of time.

In some embodiments, the computer system displays the first user interface object in a first view of the three-dimensional environment and displays background content behind the first user interface object in the three-dimensional environment. In some implementations, the first user interface object includes a rear portion having an appearance selected based on an appearance of background content located behind the first portion of the first user interface object in the three-dimensional environment, and content displayed as embedded in the first user interface object between the rear portion of the first user interface object and the front portion of the first user interface object. In some embodiments, in response to a request to move content, the computer system moves the content within the first user interface object while the content remains between the front and back of the first user interface object, and displays a visual effect applied to the content based on the simulated thickness of the first user interface object to simulate an appearance of the content reflecting, refracting, or wrapping, or otherwise bending around an edge of the simulated thickness of the first user interface object.

In some embodiments, the computer system simultaneously displays a user interface including a view of the three-dimensional environment and a simulated three-dimensional object in the three-dimensional environment, the simulated three-dimensional object being displayed with an appearance corresponding to the respective simulated material. A first region of the simulated three-dimensional object is displayed as a recessed region and is associated with a first user interface element. In some implementations, a computer system detects a first input directed to a first user interface element and, in response to detecting the first input directed to the first user interface element, performs a respective operation associated with the first user interface element in a user interface. In some embodiments, the edges of the simulated three-dimensional object are displayed with a visual effect that specular reflection simulates the reflection of light from the three-dimensional environment from the edges. In some embodiments, specular reflection is based on one or more simulated light sources and/or one or more physical light sources in a physical environment corresponding to the displayed three-dimensional environment. In some embodiments, the appearance of specular reflection changes as the simulated three-dimensional object moves in the three-dimensional environment, as the user's point of view relative to the simulated three-dimensional object changes, and/or as the illumination in the three-dimensional environment changes.

In some embodiments, a computer system displays a user interface that includes a view of a three-dimensional environment and controls associated with respective surfaces in the three-dimensional environment. In some implementations, a first set of graphical elements that are part of the control are associated with a first layer of user interface elements; a second set of graphical elements that are part of the control are associated with a second layer of user interface elements. In some embodiments, the control is displayed with an appearance that indicates a respective degree of separation between the first layer and the second layer (e.g., and optionally one or more other layers of the control). In some embodiments, in response to detecting a first input corresponding to a user focusing attention on a user interface element, the computer system changes an appearance of the user interface element to indicate a different degree of separation between the first layer and the second layer (e.g., and optionally one or more other layers of the control). In some embodiments, the degree of separation between layers of the control depends on the type and degree of user interaction with the control. In some implementations, the degree of separation between layers of the control changes in response to user interaction making a predefined gesture, and the change in the degree of separation is reversed in response to user interaction reversing the predefined gesture. In some embodiments, one or more graphical elements of the control are animated in response to user interaction.

In some embodiments, the computer system displays selectable objects that can be activated to perform operations in the three-dimensional environment as part of user interface objects in the viewable three-dimensional environment. In some embodiments, in response to a user input indicating that the user is ready to interact with the selectable object, the computer system changes (e.g., increases) the thickness of the selectable object while optionally changing one or more additional visual properties of the selectable object and/or leaving one or more visual properties of the selectable object unchanged. Changing the thickness of the selectable object in response to an input indicating the user's intent to interact with the selectable object helps the user to know which portion of the current environment has focus for further interaction.

In some embodiments, the computer system displays a computer-generated user interface object overlaid on a portion of the view of the three-dimensional environment, and the appearance of the computer-generated user interface object is based on a blurred representation of an underlying portion of the view of the three-dimensional environment. In some embodiments, the computer system visually distinguishes a first region of the computer-generated user interface object from a second region of the computer-generated user interface object by applying a different (e.g., mutually exclusive) transformation to a different (e.g., mutually exclusive) region of the blurred representation. In some embodiments, the computer system updates the appearance of the computer-generated user interface object as the appearance of the underlying portion of the view of the three-dimensional environment changes, including by applying different transformations to different regions of the blurred representation of the changed underlying portion of the view, respectively. Applying different transformations to different regions of a computer-generated object visually distinguishes some regions from other regions to indicate and/or emphasize certain properties of the regions, such as whether the regions are interactive and/or whether the regions convey semantic meaning (e.g., through glyphs and/or text), and in some cases improve legibility of the regions.

Fig. 1-6 provide a description of an exemplary computer system for providing an XR experience to a user. The user interfaces in fig. 7A to 7Z are used to illustrate the processes in fig. 8 to 14.

The processes described below enhance operability of a device and make a user-device interface more efficient (e.g., by helping a user provide appropriate input and reducing user errors in operating/interacting with the device) through various techniques, including by providing improved visual, audio, and/or tactile feedback to the user, improving user safety and reducing motion sickness typically associated with XR experiences, reducing the number of inputs required to perform operations, providing additional control options without cluttering the user interface with additional display controls, performing operations when a set of conditions has been met without further user input, reducing computational burden associated with displaying the user interface, improving privacy and/or security, providing a more diverse, detailed, and/or realistic user experience while conserving storage space, and/or additional techniques. These techniques also reduce power usage and extend battery life of the device by enabling a user to use the device faster and more efficiently. Saving battery power and thus weight, improves the ergonomics of the device. These techniques also enable real-time communication, allow fewer and/or less accurate sensors to be used, resulting in a more compact, lighter, and cheaper device, and enable the device to be used under a variety of lighting conditions. These techniques reduce energy usage and thus heat emitted by the device, which is particularly important for wearable devices because if the operating parameters of the device components are fully compliant with the operating parameters of the device, the user may feel uncomfortable wearing if excessive heat is generated.

Furthermore, in a method described herein in which one or more steps are dependent on one or more conditions having been met, it should be understood that the method may be repeated in multiple iterations such that during the iteration, all conditions that determine steps in the method have been met in different iterations of the method. For example, if a method requires performing a first step (if a condition is met) and performing a second step (if a condition is not met), one of ordinary skill will know that the stated steps are repeated until both the condition and the condition are not met (not sequentially). Thus, a method described as having one or more steps depending on one or more conditions having been met may be rewritten as a method that repeats until each of the conditions described in the method have been met. However, this does not require the system or computer-readable medium to claim that the system or computer-readable medium contains instructions for performing the contingent operation based on the satisfaction of the corresponding condition or conditions, and thus is able to determine whether the contingent situation has been met without explicitly repeating the steps of the method until all conditions to decide on steps in the method have been met. It will also be appreciated by those of ordinary skill in the art that, similar to a method with optional steps, a system or computer readable storage medium may repeat the steps of the method as many times as necessary to ensure that all optional steps have been performed.

In some embodiments, as shown in fig. 1, an XR experience is provided to a user via an operating environment 100 comprising a computer system 101. The computer system 101 includes a controller 110 (e.g., a processor or remote server of a portable electronic device), a display generation component 120 (e.g., a Head Mounted Device (HMD), a display, a projector, a touch screen, etc.), one or more input devices 125 (e.g., an eye tracking device 130, a hand tracking device 140, other input devices 150), one or more output devices 155 (e.g., a speaker 160, a haptic output generator 170, and other output devices 180), one or more sensors 190 (e.g., an image sensor, a light sensor, a depth sensor, a haptic sensor, an orientation sensor, a proximity sensor, a temperature sensor, a position sensor, a motion sensor, a speed sensor, etc.), and optionally one or more peripheral devices 195 (e.g., a household appliance, a wearable device, etc.). In some implementations, one or more of the input device 125, the output device 155, the sensor 190, and the peripheral device 195 are integrated with the display generating component 120 (e.g., in a head-mounted device or a handheld device).

In describing an XR experience, various terms are used to refer differently to several related but different environments that a user may sense and/or interact with (e.g., interact with inputs detected by computer system 101 that generated the XR experience, such inputs causing the computer system that generated the XR experience to generate audio, visual, and/or tactile feedback corresponding to various inputs provided to computer system 101). The following are a subset of these terms:

Physical environment: a physical environment refers to a physical world in which people can sense and/or interact without the assistance of an electronic system. Physical environments such as physical parks include physical objects such as physical trees, physical buildings, and physical people. People can directly sense and/or interact with a physical environment, such as by visual, tactile, auditory, gustatory, and olfactory.

And (3) augmented reality: conversely, an augmented reality (XR) environment refers to a fully or partially simulated environment in which people perceive and/or interact via an electronic system. In XR, a subset of the physical movements of the person, or a representation thereof, is tracked, and in response, one or more characteristics of one or more virtual objects simulated in the XR environment are adjusted in a manner consistent with at least one physical law. For example, an XR system may detect a person's head rotation and, in response, adjust the graphical content and sound field presented to the person in a manner similar to the manner in which such views and sounds change in a physical environment. In some cases (e.g., for reachability reasons), the adjustment of the characteristics of the virtual object in the XR environment may be made in response to a representation of the physical motion (e.g., a voice command). A person may utilize any of his sensations to sense and/or interact with XR objects, including vision, hearing, touch, taste, and smell. For example, a person may sense and/or interact with audio objects that create a 3D or spatial audio environment that provides perception of a point audio source in 3D space. As another example, an audio object may enable audio transparency that selectively introduces environmental sounds from a physical environment with or without computer generated audio. In some XR environments, a person may sense and/or interact with only audio objects.

Examples of XRs include virtual reality and mixed reality.

Virtual reality: a Virtual Reality (VR) environment refers to a simulated environment designed to be based entirely on computer-generated sensory input for one or more senses. The VR environment includes a plurality of virtual objects that a person can sense and/or interact with. For example, computer-generated images of trees, buildings, and avatars representing people are examples of virtual objects. A person may sense and/or interact with virtual objects in the VR environment through a simulation of the presence of the person within the computer-generated environment and/or through a simulation of a subset of the physical movements of the person within the computer-generated environment.

Mixed reality: in contrast to VR environments designed to be based entirely on computer-generated sensory input, a Mixed Reality (MR) environment refers to a simulated environment designed to introduce sensory input from a physical environment or a representation thereof in addition to including computer-generated sensory input (e.g., virtual objects). On a virtual continuum, a mixed reality environment is any condition between, but not including, a full physical environment as one end and a virtual reality environment as the other end. In some MR environments, the computer-generated sensory input may be responsive to changes in sensory input from the physical environment. In addition, some electronic systems for rendering MR environments may track the position and/or orientation relative to the physical environment to enable virtual objects to interact with real objects (i.e., physical objects or representations thereof from the physical environment). For example, the system may cause the motion such that the virtual tree appears to be stationary relative to the physical ground.

Examples of mixed reality include augmented reality and augmented virtualization.

Augmented reality: an Augmented Reality (AR) environment refers to a simulated environment in which one or more virtual objects are superimposed over a physical environment or a representation of a physical environment. For example, an electronic system for presenting an AR environment may have a transparent or translucent display through which a person may directly view the physical environment. The system may be configured to present the virtual object on a transparent or semi-transparent display such that a person perceives the virtual object superimposed over the physical environment with the system. Alternatively, the system may have an opaque display and one or more imaging sensors that capture images or videos of the physical environment, which are representations of the physical environment. The system combines the image or video with the virtual object and presents the composition on an opaque display. A person utilizes the system to indirectly view the physical environment via an image or video of the physical environment and perceive a virtual object superimposed over the physical environment. As used herein, video of a physical environment displayed on an opaque display is referred to as "pass-through video," meaning that the system captures images of the physical environment using one or more image sensors and uses those images when rendering an AR environment on the opaque display. Further alternatively, the system may have a projection system that projects the virtual object into the physical environment, for example as a hologram or on a physical surface, such that a person perceives the virtual object superimposed on top of the physical environment with the system. An augmented reality environment also refers to a simulated environment in which a representation of a physical environment is transformed by computer-generated sensory information. For example, in providing a passthrough video, the system may transform one or more sensor images to apply a selected viewing angle (e.g., a viewpoint) that is different from the viewing angle captured by the imaging sensor. As another example, the representation of the physical environment may be transformed by graphically modifying (e.g., magnifying) portions thereof such that the modified portions may be representative but not real versions of the original captured image. For another example, the representation of the physical environment may be transformed by graphically eliminating or blurring portions thereof.

Enhanced virtualization: enhanced virtual (AV) environment refers to a simulated environment in which a virtual environment or computer-generated environment incorporates one or more sensory inputs from a physical environment. The sensory input may be a representation of one or more characteristics of the physical environment. For example, an AV park may have virtual trees and virtual buildings, but the face of a person is realistically reproduced from an image taken of a physical person. As another example, the virtual object may take the shape or color of a physical object imaged by one or more imaging sensors. For another example, the virtual object may employ shadows that conform to the positioning of the sun in the physical environment.

Viewpoint-locked virtual object: when the computer system displays the virtual object at the same location and/or position in the user's viewpoint, the virtual object is viewpoint-locked even if the user's viewpoint is offset (e.g., changed). In embodiments in which the computer system is a head-mounted device, the user's point of view is locked to the forward direction of the user's head (e.g., when the user looks directly in front, the user's point of view is at least a portion of the user's field of view); thus, the user's point of view remains fixed without moving the user's head, even when the user's gaze is offset. In embodiments in which the computer system has a display generating component (e.g., a display screen) that is repositionable relative to the user's head, the user's point of view is an augmented reality view presented to the user on the display generating component of the computer system. For example, a viewpoint-locked virtual object displayed in the upper left corner of the user's viewpoint continues to be displayed in the upper left corner of the user's viewpoint when the user's viewpoint is in a first orientation (e.g., the user's head faces north), even when the user's viewpoint changes to a second orientation (e.g., the user's head faces west). In other words, the position and/or orientation of the virtual object in which the viewpoint lock is displayed in the viewpoint of the user is independent of the position and/or orientation of the user in the physical environment. In embodiments in which the computer system is a head-mounted device, the user's point of view is locked to the orientation of the user's head, such that the virtual object is also referred to as a "head-locked virtual object.

Environment-locked visual object: when a computer system displays a virtual object at a location and/or position in a user's point of view, the virtual object is environment-locked (alternatively, "world-locked"), the location and/or position being based on (e.g., selected and/or anchored to) a location and/or object in a three-dimensional environment (e.g., a physical environment or virtual environment) with reference to the location and/or object. As the user's point of view moves, the position and/or object in the environment relative to the user's point of view changes, which results in the environment-locked virtual object being displayed at a different position and/or location in the user's point of view. For example, an environmentally locked virtual object that locks onto a tree immediately in front of the user is displayed at the center of the user's viewpoint. When the user's viewpoint is shifted to the right (e.g., the user's head is turned to the right) such that the tree is now to the left of center in the user's viewpoint (e.g., the tree positioning in the user's viewpoint is shifted), the environmentally locked virtual object that is locked onto the tree is displayed to the left of center in the user's viewpoint. In other words, the position and/or orientation at which the environment-locked virtual object is displayed in the user's viewpoint depends on the position and/or orientation of the object and/or the position at which the virtual object is locked in the environment. In some embodiments, the computer system uses a stationary frame of reference (e.g., a coordinate system anchored to a fixed location and/or object in the physical environment) in order to determine the location of the virtual object that displays the environmental lock in the viewpoint of the user. The environment-locked virtual object may be locked to a stationary portion of the environment (e.g., a floor, wall, table, or other stationary object), or may be locked to a movable portion of the environment (e.g., a representation of a vehicle, animal, person, or even a portion of a user's body such as a user's hand, wrist, arm, or foot that moves independent of the user's point of view) such that the virtual object moves as the point of view or the portion of the environment moves to maintain a fixed relationship between the virtual object and the portion of the environment.

In some implementations, the environmentally or view-locked virtual object exhibits an inert follow-up behavior that reduces or delays movement of the environmentally or view-locked virtual object relative to movement of a reference point that the virtual object follows. In some embodiments, the computer system intentionally delays movement of the virtual object when detecting movement of a reference point (e.g., a portion of the environment, a viewpoint, or a point fixed relative to the viewpoint, such as a point between 5cm and 300cm from the viewpoint) that the virtual object is following while exhibiting inert follow-up behavior. For example, when a reference point (e.g., the portion or viewpoint of the environment) moves at a first speed, the virtual object is moved by the device to remain locked to the reference point, but moves at a second speed that is slower than the first speed (e.g., until the reference point stops moving or slows down, at which point the virtual object begins to catch up with the reference point). In some embodiments, when the virtual object exhibits inert follow-up behavior, the device ignores small movements of the reference point (e.g., ignores movements of the reference point below a threshold amount of movement, such as movements of 0 to 5 degrees or movements of 0 to 50 cm). For example, when a reference point (e.g., the portion or point of view of the environment to which the virtual object is locked) moves a first amount, the distance between the reference point and the virtual object increases (e.g., because the virtual object is being displayed so as to maintain a fixed or substantially fixed position relative to the portion of the environment or point of view other than the reference point to which the virtual object is locked), and when the reference point (e.g., the portion or point of view of the environment to which the virtual object is locked) moves a second amount that is greater than the first amount, the distance between the reference point and the virtual object initially increases (e.g., because the virtual object is being displayed so as to maintain a fixed or substantially fixed position relative to the portion of the environment other than the point of view or point to which the virtual object is locked), and then decreases as the amount of movement of the reference point increases above a threshold (e.g., an "inertia following" threshold) because the virtual object is moved by the computer system to maintain a fixed or substantially fixed position relative to the reference point. In some embodiments, maintaining a substantially fixed position of the virtual object relative to the reference point includes the virtual object being displayed within a threshold distance (e.g., 1cm, 2cm, 3cm, 5cm, 15cm, 20cm, 50 cm) of the reference point in one or more dimensions (e.g., up/down, left/right, and/or forward/backward relative to the position of the reference point).

Hardware: there are many different types of electronic systems that enable a person to sense and/or interact with various XR environments. Examples include head-mounted systems, projection-based systems, head-up displays (HUDs), vehicle windshields integrated with display capabilities, windows integrated with display capabilities, displays formed as lenses designed for placement on a person's eyes (e.g., similar to contact lenses), headphones/earphones, speaker arrays, input systems (e.g., wearable or handheld controllers with or without haptic feedback), smartphones, tablets, and desktop/laptop computers. The head-mounted system may have one or more speakers and an integrated opaque display. alternatively, the head-mounted system may be configured to accept an external opaque display (e.g., a smart phone). The head-mounted system may incorporate one or more imaging sensors for capturing images or video of the physical environment and/or one or more microphones for capturing audio of the physical environment. The head-mounted system may have a transparent or translucent display instead of an opaque display. The transparent or translucent display may have a medium through which light representing an image is directed to the eyes of a person. The display may utilize digital light projection, OLED, LED, uLED, liquid crystal on silicon, laser scanning light sources, or any combination of these techniques. The medium may be an optical waveguide, a holographic medium, an optical combiner, an optical reflector, or any combination thereof. In one embodiment, the transparent or translucent display may be configured to selectively become opaque. Projection-based systems may employ retinal projection techniques that project a graphical image onto a person's retina. The projection system may also be configured to project the virtual object into the physical environment, for example as a hologram or on a physical surface. In some embodiments, the controller 110 is configured to manage and coordinate the XR experience of the user. In some embodiments, controller 110 includes suitable combinations of software, firmware, and/or hardware. The controller 110 is described in more detail below with reference to fig. 2. In some implementations, the controller 110 is a computing device that is in a local or remote location relative to the scene 105 (e.g., physical environment). For example, the controller 110 is a local server located within the scene 105. As another example, the controller 110 is a remote server (e.g., cloud server, central server, etc.) located outside of the scene 105. In some implementations, the controller 110 is communicatively coupled with the display generation component 120 (e.g., HMD, display, projector, touch-screen, etc.) via one or more wired or wireless communication channels 144 (e.g., bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). in another example, the controller 110 is included within a housing (e.g., a physical enclosure) of the display generation component 120 (e.g., an HMD or portable electronic device including a display and one or more processors, etc.), one or more of the input devices 125, one or more of the output devices 155, one or more of the sensors 190, and/or one or more of the peripheral devices 195, or shares the same physical housing or support structure with one or more of the above.

In some embodiments, display generation component 120 is configured to provide an XR experience (e.g., at least a visual component of the XR experience) to a user. In some embodiments, display generation component 120 includes suitable combinations of software, firmware, and/or hardware. The display generating section 120 is described in more detail below with respect to fig. 3. In some embodiments, the functionality of the controller 110 is provided by and/or combined with the display generating component 120.

According to some embodiments, display generation component 120 provides an XR experience to a user when the user is virtually and/or physically present within scene 105.

In some embodiments, the display generating component is worn on a portion of the user's body (e.g., on his/her head, on his/her hand, etc.). As such, display generation component 120 includes one or more XR displays provided for displaying XR content. For example, in various embodiments, the display generation component 120 encloses a field of view of a user. In some embodiments, display generation component 120 is a handheld device (such as a smart phone or tablet computer) configured to present XR content, and the user holds the device with a display facing the user's field of view and a camera facing scene 105. In some embodiments, the handheld device is optionally placed within a housing that is worn on the head of the user. In some embodiments, the handheld device is optionally placed on a support (e.g., tripod) in front of the user. In some embodiments, display generation component 120 is an XR room, housing, or room configured to present XR content, wherein the user does not wear or hold display generation component 120. Many of the user interfaces described with reference to one type of hardware for displaying XR content (e.g., a handheld device or a device on a tripod) may be implemented on another type of hardware for displaying XR content (e.g., an HMD or other wearable computing device). For example, a user interface showing interactions with XR content triggered based on interactions occurring in a space in front of a handheld device or a tripod-mounted device may similarly be implemented with an HMD, where the interactions occur in the space in front of the HMD and responses to the XR content are displayed via the HMD. Similarly, a user interface showing interaction with XR content triggered based on movement of a handheld device or tripod-mounted device relative to a physical environment (e.g., a scene 105 or a portion of a user's body (e.g., a user's eye, head, or hand)) may similarly be implemented with an HMD, where the movement is caused by movement of the HMD relative to the physical environment (e.g., the scene 105 or a portion of the user's body (e.g., a user's eye, head, or hand)).

While relevant features of the operating environment 100 are shown in fig. 1, those of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary embodiments disclosed herein.

Fig. 2 is a block diagram of an example of a controller 110 according to some embodiments. While certain specific features are shown, those of ordinary skill in the art will appreciate from the disclosure that various other features are not shown for the sake of brevity and so as not to obscure more pertinent aspects of the embodiments disclosed herein. To this end, as a non-limiting example, in some embodiments, the controller 110 includes one or more processing units 202 (e.g., microprocessors, application Specific Integrated Circuits (ASICs), field Programmable Gate Arrays (FPGAs), graphics Processing Units (GPUs), central Processing Units (CPUs), processing cores, etc.), one or more input/output (I/O) devices 206, one or more communication interfaces 208 (e.g., universal Serial Bus (USB), IEEE 802.3x, IEEE 802.11x, IEEE 802.16x, global system for mobile communications (GSM), code Division Multiple Access (CDMA), time Division Multiple Access (TDMA), global Positioning System (GPS), infrared (IR), bluetooth, ZIGBEE, and/or similar types of interfaces), one or more programming (e.g., I/O) interfaces 210, memory 220, and one or more communication buses 204 for interconnecting these components and various other components.

In some embodiments, one or more of the communication buses 204 include circuitry that interconnects and controls communications between system components. In some embodiments, the one or more I/O devices 206 include at least one of a keyboard, a mouse, a touchpad, a joystick, one or more microphones, one or more speakers, one or more image sensors, one or more displays, and the like.

Memory 220 includes high-speed random access memory such as Dynamic Random Access Memory (DRAM), static Random Access Memory (SRAM), double data rate random access memory (DDR RAM), or other random access solid state memory devices. In some embodiments, memory 220 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 220 optionally includes one or more storage devices located remotely from the one or more processing units 202. Memory 220 includes a non-transitory computer-readable storage medium. In some embodiments, memory 220 or a non-transitory computer readable storage medium of memory 220 stores the following programs, modules, and data structures, or a subset thereof, including optional operating system 230 and XR experience module 240.

Operating system 230 includes instructions for handling various basic system services and for performing hardware-related tasks. In some embodiments, XR experience module 240 is configured to manage and coordinate single or multiple XR experiences of one or more users (e.g., single XR experiences of one or more users, or multiple XR experiences of a respective group of one or more users). To this end, in various embodiments, the XR experience module 240 includes a data acquisition unit 242, a tracking unit 244, a coordination unit 246, and a data transmission unit 248.

In some embodiments, the data acquisition unit 242 is configured to acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from at least the display generation component 120 of fig. 1, and optionally from one or more of the input device 125, the output device 155, the sensor 190, and/or the peripheral device 195. For this purpose, in various embodiments, the data acquisition unit 242 includes instructions and/or logic for instructions as well as heuristics and metadata for heuristics.

In some embodiments, tracking unit 244 is configured to map scene 105 and track at least the location/position of display generation component 120 relative to scene 105 of fig. 1, and optionally the location of one or more of input device 125, output device 155, sensor 190, and/or peripheral device 195. For this purpose, in various embodiments, tracking unit 244 includes instructions and/or logic for instructions as well as heuristics and metadata for heuristics. In some embodiments, tracking unit 244 includes a hand tracking unit 245 and/or an eye tracking unit 243. In some embodiments, the hand tracking unit 245 is configured to track the location/position of one or more portions of the user's hand, and/or the motion of one or more portions of the user's hand relative to the scene 105 of fig. 1, relative to the display generating component 120, and/or relative to a coordinate system defined relative to the user's hand. The hand tracking unit 245 is described in more detail below with respect to fig. 4. In some embodiments, the eye tracking unit 243 is configured to track the positioning or movement of the user gaze (or more generally, the user's eyes, face, or head) relative to the scene 105 (e.g., relative to the physical environment and/or relative to the user (e.g., the user's hand)) or relative to XR content displayed via the display generating component 120. The eye tracking unit 243 is described in more detail below with respect to fig. 5.

In some embodiments, coordination unit 246 is configured to manage and coordinate XR experiences presented to a user by display generation component 120, and optionally by one or more of output device 155 and/or peripheral device 195. For this purpose, in various embodiments, coordination unit 246 includes instructions and/or logic for instructions as well as heuristics and metadata for heuristics.

In some embodiments, the data transmission unit 248 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the display generation component 120, and optionally to one or more of the input device 125, the output device 155, the sensor 190, and/or the peripheral device 195. For this purpose, in various embodiments, the data transmission unit 248 includes instructions and/or logic for instructions as well as heuristics and metadata for heuristics.

While the data acquisition unit 242, tracking unit 244 (e.g., including the eye tracking unit 243 and hand tracking unit 245), coordination unit 246, and data transmission unit 248 are shown as residing on a single device (e.g., controller 110), it should be understood that in other embodiments, any combination of the data acquisition unit 242, tracking unit 244 (e.g., including the eye tracking unit 243 and hand tracking unit 245), coordination unit 246, and data transmission unit 248 may reside in a single computing device.

Furthermore, FIG. 2 is a functional description of various features that may be present in a particular implementation, as opposed to a schematic of the embodiments described herein. As will be appreciated by one of ordinary skill in the art, the individually displayed items may be combined and some items may be separated. For example, some of the functional blocks shown separately in fig. 2 may be implemented in a single block, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of particular functions, and how features are allocated among them, will vary depending upon the particular implementation, and in some embodiments, depend in part on the particular combination of hardware, software, and/or firmware selected for a particular implementation.

Fig. 3 is a block diagram of an example of display generation component 120 according to some embodiments. While certain specific features are shown, those of ordinary skill in the art will appreciate from the disclosure that various other features are not shown for the sake of brevity and so as not to obscure more pertinent aspects of the embodiments disclosed herein. For this purpose, as a non-limiting example, in some embodiments, display generation component 120 (e.g., HMD) includes one or more processing units 302 (e.g., microprocessors, ASIC, FPGA, GPU, CPU, processing cores, etc.), one or more input/output (I/O) devices and sensors 306, one or more communication interfaces 308 (e.g., ,USB、FIREWIRE、THUNDERBOLT、IEEE 802.3x、IEEE 802.11x、IEEE 802.16x、GSM、CDMA、TDMA、GPS、IR、BLUETOOTH、ZIGBEE and/or similar types of interfaces), one or more programming (e.g., I/O) interfaces 310, one or more XR displays 312, one or more optional internally and/or externally facing image sensors 314, memory 320, and one or more communication buses 304 for interconnecting these components and various other components.

In some embodiments, one or more communication buses 304 include circuitry for interconnecting and controlling communications between various system components. In some embodiments, the one or more I/O devices and sensors 306 include an Inertial Measurement Unit (IMU), an accelerometer, a gyroscope, a thermometer, one or more physiological sensors (e.g., blood pressure monitor, heart rate monitor, blood oxygen sensor, blood glucose sensor, etc.), one or more microphones, one or more speakers, a haptic engine, and/or one or more depth sensors (e.g., structured light, time of flight, etc.), and/or the like.

In some embodiments, one or more XR displays 312 are configured to provide an XR experience to a user. In some embodiments, one or more XR displays 312 correspond to holographic, digital Light Processing (DLP), liquid Crystal Displays (LCD), liquid crystal on silicon (LCoS), organic light emitting field effect transistors (OLET), organic Light Emitting Diodes (OLED), surface conduction electron emission displays (SED), field Emission Displays (FED), quantum dot light emitting diodes (QD-LED), microelectromechanical systems (MEMS), and/or similar display types. In some embodiments, one or more XR displays 312 correspond to diffractive, reflective, polarizing, holographic, etc. waveguide displays. For example, the display generation component 120 (e.g., HMD) includes a single XR display. As another example, display generation component 120 includes an XR display for each eye of the user. In some embodiments, one or more XR displays 312 are capable of presenting MR and VR content. In some implementations, one or more XR displays 312 can present MR or VR content.

In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's face including the user's eyes (and may be referred to as an eye tracking camera). In some embodiments, the one or more image sensors 314 are configured to acquire image data corresponding to at least a portion of the user's hand and optionally the user's arm (and may be referred to as a hand tracking camera). In some implementations, the one or more image sensors 314 are configured to face forward in order to acquire image data corresponding to a scene that a user would see in the absence of the display generating component 120 (e.g., HMD) (and may be referred to as a scene camera). The one or more optional image sensors 314 may include one or more RGB cameras (e.g., with Complementary Metal Oxide Semiconductor (CMOS) image sensors or Charge Coupled Device (CCD) image sensors), one or more Infrared (IR) cameras, and/or one or more event-based cameras, etc.

Memory 320 includes high-speed random access memory such as DRAM, SRAM, DDR RAM or other random access solid state memory devices. In some embodiments, memory 320 includes non-volatile memory, such as one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, or other non-volatile solid state storage devices. Memory 320 optionally includes one or more storage devices located remotely from the one or more processing units 302. Memory 320 includes a non-transitory computer-readable storage medium. In some embodiments, memory 320 or a non-transitory computer readable storage medium of memory 320 stores the following programs, modules, and data structures, or a subset thereof, including optional operating system 330 and XR presentation module 340.

Operating system 330 includes processes for handling various basic system services and for performing hardware-related tasks. In some embodiments, XR presentation module 340 is configured to present XR content to a user via one or more XR displays 312. For this purpose, in various embodiments, the XR presentation module 340 includes a data acquisition unit 342, an XR presentation unit 344, an XR map generation unit 346, and a data transmission unit 348.

In some embodiments, the data acquisition unit 342 is configured to at least acquire data (e.g., presentation data, interaction data, sensor data, location data, etc.) from the controller 110 of fig. 1. For this purpose, in various embodiments, the data acquisition unit 342 includes instructions and/or logic for instructions and heuristics and metadata for heuristics.

In some embodiments, XR presentation unit 344 is configured to present XR content via one or more XR displays 312. For this purpose, in various embodiments, XR presentation unit 344 includes instructions and/or logic for instructions and heuristics and metadata for heuristics.

In some embodiments, XR map generation unit 346 is configured to generate an XR map based on the media content data (e.g., a 3D map of a mixed reality scene or a map of a physical environment in which computer-generated objects may be placed to generate an augmented reality). For this purpose, in various embodiments, XR map generation unit 346 includes instructions and/or logic for the instructions as well as heuristics and metadata for the heuristics.

In some embodiments, the data transmission unit 348 is configured to transmit data (e.g., presentation data, location data, etc.) to at least the controller 110, and optionally one or more of the input device 125, the output device 155, the sensor 190, and/or the peripheral device 195. For this purpose, in various embodiments, the data transmission unit 348 includes instructions and/or logic for instructions and heuristics and metadata for heuristics.

Although the data acquisition unit 342, the XR presentation unit 344, the XR map generation unit 346, and the data transmission unit 348 are shown as residing on a single device (e.g., the display generation component 120 of fig. 1), it should be understood that in other embodiments, any combination of the data acquisition unit 342, the XR presentation unit 344, the XR map generation unit 346, and the data transmission unit 348 may be located in separate computing devices.

Furthermore, fig. 3 is used more as a functional description of various features that may be present in a particular embodiment, as opposed to a schematic of the embodiments described herein. As will be appreciated by one of ordinary skill in the art, the individually displayed items may be combined and some items may be separated. For example, some of the functional blocks shown separately in fig. 3 may be implemented in a single block, and the various functions of a single functional block may be implemented by one or more functional blocks in various embodiments. The actual number of modules and the division of particular functions, and how features are allocated among them, will vary depending upon the particular implementation, and in some embodiments, depend in part on the particular combination of hardware, software, and/or firmware selected for a particular implementation.

Fig. 4 is a schematic illustration of an exemplary embodiment of a hand tracking device 140. In some embodiments, the hand tracking device 140 (fig. 1) is controlled by the hand tracking unit 245 (fig. 2) to track the position/location of one or more portions of the user's hand, and/or the movement of one or more portions of the user's hand relative to the scene 105 of fig. 1 (e.g., relative to a portion of the physical environment surrounding the user, relative to the display generating component 120, or relative to a portion of the user (e.g., the user's face, eyes, or head), and/or relative to a coordinate system defined relative to the user's hand). In some implementations, the hand tracking device 140 is part of the display generation component 120 (e.g., embedded in or attached to a head-mounted device). In some embodiments, the hand tracking device 140 is separate from the display generation component 120 (e.g., in a separate housing or attached to a separate physical support structure).

In some implementations, the hand tracking device 140 includes an image sensor 404 (e.g., one or more IR cameras, 3D cameras, depth cameras, and/or color cameras, etc.) that captures three-dimensional scene information including at least a human user's hand 406. The image sensor 404 captures the hand image with sufficient resolution to enable the fingers and their respective locations to be distinguished. The image sensor 404 typically captures images of other parts of the user's body, and possibly also all parts of the body, and may have a zoom capability or a dedicated sensor with increased magnification to capture images of the hand with a desired resolution. In some implementations, the image sensor 404 also captures 2D color video images of the hand 406 and other elements of the scene. In some implementations, the image sensor 404 is used in conjunction with other image sensors to capture the physical environment of the scene 105, or as an image sensor that captures the physical environment of the scene 105. In some embodiments, the image sensor 404, or a portion thereof, is positioned relative to the user or the user's environment in a manner that uses the field of view of the image sensor to define an interaction space in which hand movements captured by the image sensor are considered input to the controller 110.

In some embodiments, the image sensor 404 outputs a sequence of frames containing 3D mapping data (and, in addition, possible color image data) to the controller 110, which extracts high-level information from the mapping data. This high-level information is typically provided via an Application Program Interface (API) to an application program running on the controller, which drives the display generating component 120 accordingly. For example, a user may interact with software running on the controller 110 by moving their hand 406 and/or changing their hand pose.

In some implementations, the image sensor 404 projects a speckle pattern onto a scene that includes the hand 406 and captures an image of the projected pattern. In some implementations, the controller 110 calculates 3D coordinates of points in the scene (including points on the surface of the user's hand) by triangulation based on lateral offsets of the blobs in the pattern. This approach is advantageous because it does not require the user to hold or wear any kind of beacon, sensor or other marker. The method gives the depth coordinates of points in the scene relative to a predetermined reference plane at a specific distance from the image sensor 404. In this disclosure, it is assumed that the image sensor 404 defines an orthogonal set of x-axis, y-axis, z-axis such that the depth coordinates of points in the scene correspond to the z-component measured by the image sensor. Alternatively, the image sensor 404 (e.g., a hand tracking device) may use other 3D mapping methods, such as stereoscopic imaging or time-of-flight measurements, based on single or multiple cameras or other types of sensors.

In some implementations, the hand tracking device 140 captures and processes a time series containing a depth map of the user's hand as the user moves their hand (e.g., the entire hand or one or more fingers). Software running on the image sensor 404 and/or a processor in the controller 110 processes the 3D mapping data to extract image block descriptors of the hand in these depth maps. The software may match these descriptors with image block descriptors stored in database 408 based on previous learning processes in order to estimate the pose of the hand in each frame. The pose typically includes a 3D positioning of the user's hand joints and finger tips.

The software may also analyze the trajectory of the hand and/or finger over multiple frames in the sequence to identify gestures. The pose estimation functions described herein may alternate with motion tracking functions such that image block-based pose estimation is performed only once every two (or more) frames while tracking changes used to find poses that occur on the remaining frames. Pose, motion, and gesture information are provided to an application running on the controller 110 via the APIs described above. The program may move and modify images presented on the display generation component 120, for example, in response to pose and/or gesture information, or perform other functions.

In some implementations, the gesture includes an air gesture. An air gesture is a motion of a portion of a user's body (e.g., a head, one or more arms, one or more hands, one or more fingers, and/or one or more legs) through the air that is detected without the user touching an input element (or being independent of an input element that is part of a device) that is part of a device (e.g., computer system 101, one or more input devices 125, and/or hand tracking device 140) (including a motion of the user's body relative to an absolute reference (e.g., angle of the user's arm relative to the ground or distance of the user's hand relative to the ground), movement relative to another portion of the user's body (e.g., movement of the user's hand relative to the user's shoulder, movement of one hand of the user relative to the other hand of the user, and/or movement of the user's finger relative to the other finger or portion of the hand of the user), and/or absolute movement of a portion of the user's body (e.g., a flick gesture comprising a predetermined amount and/or speed of movement of the hand in a predetermined gesture, or a shake gesture comprising a predetermined speed or amount of rotation of a portion of the user's body)).

In some embodiments, according to some embodiments, the input gestures used in the various examples and embodiments described herein include air gestures performed by movement of a user's finger relative to other fingers or portions of the user's hand for interacting with an XR environment (e.g., a virtual or mixed reality environment). In some embodiments, the air gesture is a gesture that is detected without the user touching an input element that is part of the device (or independent of an input element that is part of the device) and based on a detected movement of a portion of the user's body through the air, including a movement of the user's body relative to an absolute reference (e.g., an angle of the user's arm relative to the ground or a distance of the user's hand relative to the ground), a movement relative to another portion of the user's body (e.g., a movement of the user's hand relative to the user's shoulder, a movement of the user's hand relative to the other hand of the user, and/or a movement of the user's finger relative to the other finger or part of the hand of the user), and/or an absolute movement of a portion of the user's body (e.g., a flick gesture that includes a predetermined amount and/or speed of movement of the hand in a predetermined gesture that includes a predetermined gesture of the hand, or a shake gesture that includes a predetermined speed or amount of rotation of a portion of the user's body).

In some embodiments where the input gesture is an air gesture (e.g., in the absence of physical contact with the input device, the input device provides information to the computer system as to which user interface element is the target of the user input, such as contact with a user interface element displayed on a touch screen, or contact with a mouse or touchpad to move a cursor to the user interface element), the gesture takes into account the user's attention (e.g., gaze) to determine the target of the user input (e.g., for direct input, as described below). Thus, in embodiments involving air gestures, for example, an input gesture in combination (e.g., simultaneously) with movement of a user's finger and/or hand detects an attention (e.g., gaze) toward a user interface element to perform pinch and/or tap inputs, as described below.

In some implementations, an input gesture directed to a user interface object is performed with direct or indirect reference to the user interface object. For example, user input is performed directly on a user interface object according to performing input with a user's hand at a location corresponding to the location of the user interface object in a three-dimensional environment (e.g., as determined based on the user's current viewpoint). In some implementations, upon detecting a user's attention (e.g., gaze) to a user interface object, an input gesture is performed indirectly on the user interface object in accordance with a position of a user's hand not being at the position corresponding to the position of the user interface object in the three-dimensional environment while the user is performing the input gesture. For example, for a direct input gesture, the user can direct the user's input to the user interface object by initiating the gesture at or near a location corresponding to the display location of the user interface object (e.g., within 0.5cm, 1cm, 5cm, or within a distance between 0 and 5cm measured from the outer edge of the option or the center portion of the option). For indirect input gestures, a user can direct the user's input to a user interface object by focusing on the user interface object (e.g., by looking at the user interface object), and while focusing on an option, the user initiates an input gesture (e.g., at any location detectable by the computer system) (e.g., at a location that does not correspond to the display location of the user interface object).

In some embodiments, according to some embodiments, the input gestures (e.g., air gestures) used in the various examples and embodiments described herein include pinch inputs and tap inputs for interacting with a virtual or mixed reality environment. For example, pinch and tap inputs described below are performed as air gestures.

In some implementations, the pinch input is part of an air gesture that includes one or more of: pinch gestures, long pinch gestures, pinch and drag gestures, or double pinch gestures. For example, pinch gestures as air gestures include movements of two or more fingers of a hand to contact each other, i.e., optionally, immediately followed by interruption of contact with each other (e.g., within 0 to 1 second). A long pinch gesture, which is an air gesture, includes movement of two or more fingers of a hand into contact with each other for at least a threshold amount of time (e.g., at least 1 second) before a break in contact with each other is detected. For example, a long pinch gesture includes a user holding a pinch gesture (e.g., where two or more fingers make contact), and the long pinch gesture continues until a break in contact between the two or more fingers is detected. In some implementations, the double pinch gesture as an air gesture includes two (e.g., or more) pinch inputs (e.g., performed by the same hand) that are detected in succession with each other immediately (e.g., within a predefined period of time). For example, the user performs a first pinch input (e.g., a pinch input or a long pinch input), releases the first pinch input (e.g., breaks contact between two or more fingers), and performs a second pinch input within a predefined period of time (e.g., within 1 second or within 2 seconds) after releasing the first pinch input.

In some implementations, the pinch-and-drag gesture as an air gesture includes a pinch gesture (e.g., a pinch gesture or a long pinch gesture) that is performed in conjunction with (e.g., follows) a drag input that changes a position of a user's hand from a first position (e.g., a start position of the drag) to a second position (e.g., an end position of the drag). In some implementations, the user holds the pinch gesture while the drag input is performed, and releases the pinch gesture (e.g., opens their two or more fingers) to end the drag gesture (e.g., at the second location). In some implementations, pinch input and drag input are performed by the same hand (e.g., a user pinch two or more fingers to contact each other and move the same hand to a second position in the air with a drag gesture). In some embodiments, pinch input is performed by a first hand of the user and drag input is performed by a second hand of the user (e.g., the second hand of the user moves in the air from a first position to a second position while the user continues to pinch input with the first hand of the user, in some embodiments, input gestures that are air gestures include input performed using both hands of the user (e.g., pinch and/or tap inputs). For example, input gestures include combining with each other (e.g., for example, a first pinch gesture (e.g., a pinch input, a long pinch input, or a pinch and drag input) is performed using a first hand of a user, and a second pinch input is performed using the other hand (e.g., a second hand of the two hands of the user) in conjunction with the first hand.

In some implementations, the tap input (e.g., pointing to the user interface element) performed as an air gesture includes movement of a user's finger toward the user interface element, movement of a user's hand toward the user interface element (optionally, the user's finger extends toward the user interface element), downward movement of the user's finger (e.g., mimicking a mouse click motion or a tap on a touch screen), or other predefined movement of the user's hand. In some embodiments, a flick input performed as an air gesture is detected based on a movement characteristic of a finger or hand performing a flick gesture movement of the finger or hand away from a user's point of view and/or toward an object that is a target of the flick input, followed by an end of the movement. In some embodiments, the end of movement is detected based on a change in movement characteristics of the finger or hand performing the flick gesture (e.g., the end of movement away from the user's point of view and/or toward an object that is the target of the flick input, the reversal of the direction of movement of the finger or hand, and/or the reversal of the acceleration direction of movement of the finger or hand).

In some embodiments, the determination that the user's attention is directed to a portion of the three-dimensional environment is based on detection of gaze directed to that portion (optionally, without other conditions). In some embodiments, the portion of the three-dimensional environment to which the user's attention is directed is determined based on detecting a gaze directed to the portion of the three-dimensional environment with one or more additional conditions, such as requiring the gaze to be directed to the portion of the three-dimensional environment for at least a threshold duration (e.g., dwell duration) and/or requiring the gaze to be directed to the portion of the three-dimensional environment when the point of view of the user is within a distance threshold from the portion of the three-dimensional environment, such that the device determines the portion of the three-dimensional environment to which the user's attention is directed, wherein if one of the additional conditions is not met, the device determines that the attention is not directed to the portion of the three-dimensional environment to which the gaze is directed (e.g., until the one or more additional conditions are met).

In some embodiments, detection of the ready state configuration of the user or a portion of the user is detected by the computer system. Detection of a ready state configuration of a hand is used by a computer system as an indication that a user may be ready to interact with the computer system using one or more air gesture inputs (e.g., pinch, tap, pinch and drag, double pinch, long pinch, or other air gestures described herein) performed by the hand. For example, the ready state of the hand is determined based on whether the hand has a predetermined hand shape (e.g., a pre-pinch shape in which the thumb and one or more fingers extend and are spaced apart in preparation for making a pinch or grasp gesture, or a pre-flick in which the one or more fingers extend and the palm faces away from the user), based on whether the hand is in a predetermined position relative to the user's point of view (e.g., below the user's head and above the user's waist and extending at least 15cm, 20cm, 25cm, 30cm, or 50cm from the body), and/or based on whether the hand has moved in a particular manner (e.g., toward an area above the user's waist and in front of the user's head or away from the user's body or legs). In some implementations, the ready state is used to determine whether an interactive element of the user interface is responsive to an attention (e.g., gaze) input.

In some embodiments, the software may be downloaded to the controller 110 in electronic form, over a network, for example, or may alternatively be provided on tangible non-transitory media, such as optical, magnetic, or electronic memory media. In some embodiments, database 408 is also stored in a memory associated with controller 110. Alternatively or in addition, some or all of the described functions of the computer may be implemented in dedicated hardware, such as a custom or semi-custom integrated circuit or a programmable Digital Signal Processor (DSP). Although the controller 110 is shown in fig. 4, for example, as a separate unit from the image sensor 404, some or all of the processing functions of the controller may be performed by a suitable microprocessor and software or by dedicated circuitry within the housing of the image sensor 404 (e.g., a hand tracking device) or other devices associated with the image sensor 404. In some embodiments, at least some of these processing functions may be performed by a suitable processor integrated with display generation component 120 (e.g., in a television receiver, handheld device, or head mounted device) or with any other suitable computerized device (such as a game console or media player). The sensing functionality of the image sensor 404 may likewise be integrated into a computer or other computerized device to be controlled by the sensor output.

Fig. 4 also includes a schematic diagram of a depth map 410 captured by the image sensor 404, according to some embodiments. As described above, the depth map comprises a matrix of pixels having corresponding depth values. Pixels 412 corresponding to the hand 406 have been segmented from the background and wrist in the map. The brightness of each pixel within the depth map 410 is inversely proportional to its depth value (i.e., the measured z-distance from the image sensor 404), where the gray shade becomes darker with increasing depth. The controller 110 processes these depth values to identify and segment components of the image (i.e., a set of adjacent pixels) that have human hand features. These features may include, for example, overall size, shape, and frame-to-frame motion from a sequence of depth maps.

Fig. 4 also schematically illustrates the hand bones 414 that the controller 110 eventually extracts from the depth map 410 of the hand 406, according to some embodiments. In fig. 4, the hand skeleton 414 is superimposed over the hand background 416 that has been segmented from the original depth map. In some embodiments, key feature points of the hand and optionally on the wrist or arm connected to the hand (e.g., points corresponding to the knuckles, finger tips, palm centers, ends of the hand connected to the wrist, etc.) are identified and located on the hand bones 414. In some embodiments, the controller 110 uses the positions and movements of these key feature points on the plurality of image frames to determine a gesture performed by the hand or a current state of the hand according to some embodiments.

Fig. 5 illustrates an exemplary embodiment of the eye tracking device 130 (fig. 1). In some embodiments, eye tracking device 130 is controlled by eye tracking unit 243 (fig. 2) to track the positioning and movement of the user gaze relative to scene 105 or relative to XR content displayed via display generation component 120. In some embodiments, the eye tracking device 130 is integrated with the display generation component 120. For example, in some embodiments, when display generating component 120 is a head-mounted device (such as a headset, helmet, goggles, or glasses) or a handheld device placed in a wearable frame, the head-mounted device includes both components that generate XR content for viewing by a user and components for tracking the user's gaze with respect to the XR content. In some embodiments, the eye tracking device 130 is separate from the display generation component 120. For example, when the display generating component is a handheld device or an XR chamber, the eye tracking device 130 is optionally a device separate from the handheld device or XR chamber. In some embodiments, the eye tracking device 130 is a head mounted device or a portion of a head mounted device. In some embodiments, the head-mounted eye tracking device 130 is optionally used in combination with a display generating component that is also head-mounted or a display generating component that is not head-mounted. In some embodiments, the eye tracking device 130 is not a head mounted device and is optionally used in conjunction with a head mounted display generating component. In some embodiments, the eye tracking device 130 is not a head mounted device and optionally is part of a non-head mounted display generating component.

In some embodiments, the display generation component 120 uses a display mechanism (e.g., a left near-eye display panel and a right near-eye display panel) to display frames including left and right images in front of the user's eyes, thereby providing a 3D virtual view to the user. For example, the head mounted display generating component may include left and right optical lenses (referred to herein as eye lenses) located between the display and the user's eyes. In some embodiments, the display generation component may include or be coupled to one or more external cameras that capture video of the user's environment for display. In some embodiments, the head mounted display generating component may have a transparent or translucent display and the virtual object is displayed on the transparent or translucent display through which the user may directly view the physical environment. In some embodiments, the display generation component projects the virtual object into the physical environment. The virtual object may be projected, for example, on a physical surface or as a hologram, such that an individual uses the system to observe the virtual object superimposed over the physical environment. In this case, separate display panels and image frames for the left and right eyes may not be required.

As shown in fig. 5, in some embodiments, the eye tracking device 130 (e.g., a gaze tracking device) includes at least one eye tracking camera (e.g., an Infrared (IR) or Near Infrared (NIR) camera) and an illumination source (e.g., an IR or NIR light source, such as an array or ring of LEDs) that emits light (e.g., IR or NIR light) toward the user's eye. The eye-tracking camera may be directed toward the user's eye to receive IR or NIR light reflected directly from the eye by the light source, or alternatively may be directed toward "hot" mirrors located between the user's eye and the display panel that reflect IR or NIR light from the eye to the eye-tracking camera while allowing visible light to pass through. The eye tracking device 130 optionally captures images of the user's eyes (e.g., as a video stream captured at 60-120 frames per second (fps)), analyzes the images to generate gaze tracking information, and communicates the gaze tracking information to the controller 110. In some embodiments, both eyes of the user are tracked separately by the respective eye tracking camera and illumination source. In some embodiments, only one eye of the user is tracked by the respective eye tracking camera and illumination source.

In some embodiments, the eye tracking device 130 is calibrated using a device-specific calibration process to determine parameters of the eye tracking device for the particular operating environment 100, such as 3D geometry and parameters of LEDs, cameras, hot mirrors (if present), eye lenses, and display screens. The device-specific calibration procedure may be performed at the factory or another facility prior to delivering the AR/VR equipment to the end user. The device-specific calibration process may be an automatic calibration process or a manual calibration process. According to some embodiments, the user-specific calibration process may include an estimation of eye parameters of a specific user, such as pupil position, foveal position, optical axis, visual axis, eye distance, etc. According to some embodiments, once the device-specific parameters and the user-specific parameters are determined for the eye-tracking device 130, the images captured by the eye-tracking camera may be processed using a flash-assist method to determine the current visual axis and gaze point of the user relative to the display.

As shown in fig. 5, the eye tracking device 130 (e.g., 130A or 130B) includes an eye lens 520 and a gaze tracking system including at least one eye tracking camera 540 (e.g., an Infrared (IR) or Near Infrared (NIR) camera) positioned on a side of the user's face on which eye tracking is performed, and an illumination source 530 (e.g., an IR or NIR light source such as an array or ring of NIR Light Emitting Diodes (LEDs)) that emits light (e.g., IR or NIR light) toward the user's eyes 592. The eye-tracking camera 540 may be directed toward a mirror 550 (which reflects IR or NIR light from the eye 592 while allowing visible light to pass) located between the user's eye 592 and the display 510 (e.g., left or right display panel of a head-mounted display, or display of a handheld device, projector, etc.) (e.g., as shown in the top portion of fig. 5), or alternatively may be directed toward the user's eye 592 to receive reflected IR or NIR light from the eye 592 (e.g., as shown in the bottom portion of fig. 5).

In some implementations, the controller 110 renders AR or VR frames 562 (e.g., left and right frames for left and right display panels) and provides the frames 562 to the display 510. The controller 110 uses the gaze tracking input 542 from the eye tracking camera 540 for various purposes, such as for processing the frames 562 for display. The controller 110 optionally estimates the gaze point of the user on the display 510 based on gaze tracking input 542 acquired from the eye tracking camera 540 using a flash assist method or other suitable method. The gaze point estimated from the gaze tracking input 542 is optionally used to determine the direction in which the user is currently looking.

Several possible use cases of the current gaze direction of the user are described below and are not intended to be limiting. As an exemplary use case, the controller 110 may render virtual content differently based on the determined direction of the user's gaze. For example, the controller 110 may generate virtual content in a foveal region determined according to a current gaze direction of the user at a higher resolution than in a peripheral region. As another example, the controller may position or move virtual content in the view based at least in part on the user's current gaze direction. As another example, the controller may display particular virtual content in the view based at least in part on the user's current gaze direction. As another exemplary use case in an AR application, the controller 110 may direct an external camera used to capture the physical environment of the XR experience to focus in the determined direction. The autofocus mechanism of the external camera may then focus on an object or surface in the environment that the user is currently looking at on display 510. As another example use case, the eye lens 520 may be a focusable lens, and the controller uses the gaze tracking information to adjust the focus of the eye lens 520 such that the virtual object the user is currently looking at has the appropriate vergence to match the convergence of the user's eyes 592. The controller 110 may utilize the gaze tracking information to direct the eye lens 520 to adjust the focus such that the approaching object the user is looking at appears at the correct distance.

In some embodiments, the eye tracking device is part of a head mounted device that includes a display (e.g., display 510), two eye lenses (e.g., eye lens 520), an eye tracking camera (e.g., eye tracking camera 540), and a light source (e.g., light source 530 (e.g., IR or NIR LED)) mounted in a wearable housing. The light source emits light (e.g., IR or NIR light) toward the user's eye 592. In some embodiments, the light sources may be arranged in a ring or circle around each of the lenses, as shown in fig. 5. In some embodiments, for example, eight light sources 530 (e.g., LEDs) are arranged around each lens 520. However, more or fewer light sources 530 may be used, and other arrangements and locations of light sources 530 may be used.

In some implementations, the display 510 emits light in the visible range and does not emit light in the IR or NIR range, and thus does not introduce noise in the gaze tracking system. Note that the position and angle of the eye tracking camera 540 is given by way of example and is not intended to be limiting. In some implementations, a single eye tracking camera 540 is located on each side of the user's face. In some implementations, two or more NIR cameras 540 may be used on each side of the user's face. In some implementations, a camera 540 with a wider field of view (FOV) and a camera 540 with a narrower FOV may be used on each side of the user's face. In some implementations, a camera 540 operating at one wavelength (e.g., 850 nm) and a camera 540 operating at a different wavelength (e.g., 940 nm) may be used on each side of the user's face.

The embodiment of the gaze tracking system as shown in fig. 5 may be used, for example, in an augmented reality (e.g., including virtual reality and/or mixed reality) application to provide an augmented reality (e.g., including virtual reality, augmented reality, and/or augmented virtual) experience to a user.

Fig. 6 illustrates a flash-assisted gaze tracking pipeline in accordance with some embodiments. In some embodiments, the gaze tracking pipeline is implemented by a glint-assisted gaze tracking system (e.g., an eye tracking device 130 as shown in fig. 1 and 5). The flash-assisted gaze tracking system may maintain a tracking state. Initially, the tracking state is off or "no". When in the tracking state, the glint-assisted gaze tracking system uses previous information from a previous frame when analyzing the current frame to track pupil contours and glints in the current frame. When not in the tracking state, the glint-assisted gaze tracking system attempts to detect pupils and glints in the current frame and, if successful, initializes the tracking state to "yes" and continues with the next frame in the tracking state.

As shown in fig. 6, the gaze tracking camera may capture left and right images of the left and right eyes of the user. The captured image is then input to the gaze tracking pipeline for processing beginning at 610. As indicated by the arrow returning to element 600, the gaze tracking system may continue to capture images of the user's eyes, for example, at a rate of 60 to 120 frames per second. In some embodiments, each set of captured images may be input to a pipeline for processing. However, in some embodiments or under some conditions, not all captured frames are pipelined.

At 610, for the currently captured image, if the tracking state is yes, the method proceeds to element 640. At 610, if the tracking state is no, the image is analyzed to detect a user's pupil and glints in the image, as indicated at 620. At 630, if the pupil and glints are successfully detected, the method proceeds to element 640. Otherwise, the method returns to element 610 to process the next image of the user's eye.

At 640, if proceeding from element 610, the current frame is analyzed to track pupils and glints based in part on previous information from the previous frame. At 640, if proceeding from element 630, a tracking state is initialized based on the pupil and flash detected in the current frame. The results of the processing at element 640 are checked to verify that the results of the tracking or detection may be trusted. For example, the results may be checked to determine if the pupil and a sufficient number of flashes for performing gaze estimation are successfully tracked or detected in the current frame. If the result is unlikely to be authentic at 650, then the tracking state is set to no at element 660 and the method returns to element 610 to process the next image of the user's eye. At 650, if the result is trusted, the method proceeds to element 670. At 670, the tracking state is set to YES (if not already YES), and pupil and glint information is passed to element 680 to estimate the gaze point of the user.

Fig. 6 is intended to serve as one example of an eye tracking technique that may be used in a particular implementation. As will be appreciated by one of ordinary skill in the art, other eye tracking techniques, currently existing or developed in the future, may be used in place of or in combination with the glint-assisted eye tracking techniques described herein in computer system 101 for providing an XR experience to a user, according to various embodiments.

In this disclosure, various input methods are described with respect to interactions with a computer system. When one input device or input method is used to provide an example and another input device or input method is used to provide another example, it should be understood that each example may be compatible with and optionally utilize the input device or input method described with respect to the other example. Similarly, various output methods are described with respect to interactions with a computer system. When one output device or output method is used to provide an example and another output device or output method is used to provide another example, it should be understood that each example may be compatible with and optionally utilize the output device or output method described with respect to the other example. Similarly, the various methods are described with respect to interactions with a virtual environment or mixed reality environment through a computer system. When examples are provided using interactions with a virtual environment, and another example is provided using a mixed reality environment, it should be understood that each example may be compatible with and optionally utilize the methods described with respect to the other example. Thus, the present disclosure discloses embodiments that are combinations of features of multiple examples, without the need to list all features of the embodiments in detail in the description of each example embodiment.

User interface and associated process

Attention is now directed to embodiments of user interfaces ("UIs") and associated processes that may be implemented on a computer system in communication with a display generating component, one or more input devices, and optionally one or more cameras, such as a portable multifunction device or a head mounted device.

Fig. 7A-7Z illustrate a three-dimensional environment visible via a display generating component (e.g., display generating component 7100 or display generating component 120) of a computer system (e.g., computer system 101), as well as interactions occurring in the three-dimensional environment caused by user inputs directed to the three-dimensional environment and/or inputs received from other computer systems and/or sensors. In some implementations, the input is directed to the virtual object within the three-dimensional environment by a user gaze detected in an area occupied by the virtual object, or by a gesture performed at a location in the physical environment corresponding to the area of the virtual object. In some implementations, the input is directed to the virtual object within the three-dimensional environment by a gesture performed (e.g., optionally, at a location in the physical environment that is independent of the area of the virtual object in the three-dimensional environment) when the virtual object has an input focus (e.g., when the virtual object has been selected by a gaze input that is detected simultaneously and/or previously, by a pointer input that is detected simultaneously or previously, and/or by a gesture input that is detected simultaneously and/or previously). In some implementations, the input is directed to a virtual object within the three-dimensional environment by an input device that has positioned a focus selector object (e.g., a pointer object or a selector object) at the location of the virtual object. In some implementations, the input is directed to a virtual object within the three-dimensional environment via other means (e.g., voice and/or control buttons). In some implementations, the input is directed to the physical object or a representation of the virtual object corresponding to the physical object by user hand movement (e.g., whole hand movement in a respective gesture, movement of one portion of the user's hand relative to another portion of the hand, and/or relative movement between the hands) and/or manipulation relative to the physical object (e.g., touch, swipe, tap, open, move-toward, and/or relative movement). In some embodiments, the computer system displays some changes to the three-dimensional environment (e.g., displays additional virtual content, stops displaying existing virtual content, and/or transitions between displaying different immersion levels of visual content) based on inputs from sensors (e.g., image sensors, temperature sensors, biometric sensors, motion sensors, and/or proximity sensors) and contextual conditions (e.g., location, time, and/or presence of other people in the environment). In some embodiments, the computer system displays some changes to the three-dimensional environment (e.g., displays additional virtual content, stops displaying existing virtual content, and/or transitions between different immersion levels of displaying visual content) based on input from other computers used by other users sharing the computer-generated environment with users of the computer system (e.g., in a shared computer-generated experience, in a shared virtual environment, and/or in a shared virtual or augmented reality environment of a communication session). In some embodiments, the computer system displays some changes to the three-dimensional environment (e.g., displaying movements, deformations, and/or changes in visual characteristics of the user interface, virtual surface, user interface object, and/or virtual landscape) based on input from sensors that detect movements of other people and objects and movements of the user that may not meet the criteria of the recognized gesture input as triggering the associated operation of the computer system.

In some embodiments, the three-dimensional environment displayed or viewable via the display generating component described herein is a virtual three-dimensional environment that includes virtual objects and content at different virtual locations in the three-dimensional environment without a representation of the physical environment. In some implementations, the three-dimensional environment is a mixed reality environment that displays virtual objects at different virtual locations in the three-dimensional environment that are constrained by one or more physical aspects of the physical environment (e.g., the location and orientation of walls, floors, surfaces, the direction of gravity, time of day, and/or spatial relationships between physical objects). In some embodiments, the three-dimensional environment is an augmented reality environment that includes a representation of a physical environment. In some embodiments, the representations of the physical environment include respective representations of the physical objects and surfaces at different locations in the three-dimensional environment such that spatial relationships between the different physical objects and surfaces in the physical environment are reflected by spatial relationships between the representations of the physical objects and surfaces in the three-dimensional environment. In some embodiments, when a virtual object is placed relative to the position of a representation of a physical object and a surface in a three-dimensional environment, the virtual object appears to have a corresponding spatial relationship to the physical object and the surface in the physical environment. In some embodiments, the computer system transitions between displaying different types of environments based on user input and/or contextual conditions (e.g., transitions between rendering a computer-generated environment or experience with different levels of immersion, adjusting the relative significance of audio/visual sensory input from the virtual content and from the representation of the physical environment).

In some embodiments, the display generating component includes a transparent portion in which a representation of the physical environment is displayed or visible. In some implementations, the transparent portion of the display-generating component is a transparent or translucent (e.g., see-through) portion of the display-generating component that displays at least a portion of the physical environment (sometimes referred to as "optical transparent") around the user or within the user's field of view. For example, the transparent portion is a portion of the head-mounted display or head-up display that is made translucent (e.g., less than 50%, 40%, 30%, 20%, 15%, 10%, or 5% opacity) or transparent so that a user can view the real world around the user through it without removing the head-mounted display or moving away from the head-up display. In some embodiments, the transparent portion gradually transitions from translucent or transparent to completely opaque when displaying a virtual or mixed reality environment. In some embodiments, the passthrough portion of the display generation component displays a real-time feed (sometimes referred to as "optical passthrough") of images or videos of at least a portion of a physical environment captured by one or more cameras (e.g., a rear facing camera of a mobile device or associated with a head mounted display, or other camera that feeds image data to a computer system). In some embodiments, the one or more cameras are directed at a portion of the physical environment directly in front of the user's eyes (e.g., behind the display generating component relative to the user of the display generating component). In some embodiments, the one or more cameras are directed at a portion of the physical environment that is not directly in front of the user's eyes (e.g., in a different physical environment, or at the side or rear of the user).

In some implementations, when virtual objects are displayed at locations corresponding to the locations of one or more physical objects in the physical environment (e.g., at locations in the virtual reality environment, the mixed reality environment, or the augmented reality environment), at least some of the virtual objects are displayed in place of (e.g., in place of) a portion of the real-time view of the camera (e.g., a portion of the physical environment captured in the real-time view). In some embodiments, at least some of the virtual objects and content are projected onto a physical surface or empty space in the physical environment and are visible through the transparent portion of the display generating component (e.g., visible as part of a camera view of the physical environment or visible through a transparent or translucent portion of the display generating component). In some embodiments, at least some of the virtual objects and virtual content are displayed to overlay portions of the display and to obscure at least a portion of the view of the physical environment that is visible through the transparent or translucent portion of the display generating component.

In some embodiments, the display generation component displays a different view of the three-dimensional environment according to user input or movement that changes a viewpoint of a currently displayed view of the three-dimensional environment relative to a virtual location of the three-dimensional environment. In some implementations, when the three-dimensional environment is a virtual environment, the viewpoint moves according to a navigation or motion request (e.g., an air gesture and/or a gesture performed by movement of one portion of the hand relative to another portion of the hand) without requiring movement of the user's head, torso, and/or display generating components in the physical environment. In some embodiments, movement of the user's head and/or torso, and/or movement of the display generating component or other location-aware element of the computer system (e.g., due to the user holding the display generating component or wearing the HMD) relative to the physical environment results in corresponding movement of the viewpoint relative to the three-dimensional environment (e.g., with corresponding movement direction, movement distance, movement speed, and/or orientation changes), resulting in corresponding changes in the current display view of the three-dimensional environment. In some embodiments, when the virtual object has a preset spatial relationship with respect to the viewpoint (e.g., is anchored or fixed to the viewpoint), movement of the viewpoint with respect to the three-dimensional environment will cause movement of the virtual object with respect to the three-dimensional environment while maintaining the position of the virtual object in the field of view (e.g., the virtual object is said to be head-locked). In some embodiments, the virtual object is physically locked to the user and moves relative to the three-dimensional environment as the user moves in the physical environment as a whole (e.g., carries or wears the display generating component and/or other position sensing components of the computer system), but will not move in the three-dimensional environment in response to individual user head movements (e.g., the display generating component and/or other position sensing components of the computer system rotate about a fixed position of the user in the physical environment). In some embodiments, the virtual object is optionally locked to another portion of the user, such as the user's hand or the user's wrist, and moves in the three-dimensional environment according to movement of the portion of the user in the physical environment to maintain a preset spatial relationship between the position of the virtual object and the virtual position of the portion of the user in the three-dimensional environment. In some embodiments, the virtual object is locked to a preset portion of the field of view provided by the display generating component and moves in a three-dimensional environment according to movement of the field of view, independent of movement of the user that does not cause a change in the field of view.

In some embodiments, as shown in fig. 7A-7Z, the view of the three-dimensional environment sometimes does not include representations of the user's hands, arms, and/or wrists. In some embodiments, representations of a user's hands, arms, and/or wrists are included in a view of a three-dimensional environment. In some embodiments, the representation of the user's hand, arm, and/or wrist is included in a view of the three-dimensional environment as part of the representation of the physical environment provided via the display generating component. In some embodiments, these representations are not part of the representation of the physical environment and are captured (e.g., pointed at the user's hand, arm, and wrist by one or more cameras) and displayed separately in a three-dimensional environment independent of the current display view of the three-dimensional environment. In some embodiments, these representations include camera images captured by one or more cameras of the computer system or stylized versions of the arm, wrist, and/or hand based on information captured by the various sensors. In some embodiments, these representations replace a display of, are overlaid on, or block a view of, a portion of the representation of the physical environment. In some embodiments, when the display generating component does not provide a view of the physical environment and provides a full virtual environment (e.g., no camera view and no transparent passthrough portion), a real-time visual representation (e.g., a stylized representation or a segmented camera image) of one or both arms, wrists, and/or hands of the user is optionally still displayed in the virtual environment. In some embodiments, if no representation of the user's hand is provided in the view of the three-dimensional environment, the location corresponding to the user's hand is optionally indicated in the three-dimensional environment, for example, by changing the appearance of the virtual content (e.g., by translucency and/or simulating a change in reflectivity) at a location in the three-dimensional environment corresponding to the location of the user's hand in the physical environment. In some embodiments, the representation of the user's hand or wrist is outside of the current display view of the three-dimensional environment, while the virtual position in the three-dimensional environment corresponding to the position of the user's hand or wrist is outside of the current field of view provided via the display generating component; and responsive to the virtual position corresponding to the position of the user's hand or wrist moving within the current field of view due to movement of the display generating component, the user's hand or wrist, the user's head, and/or the user as a whole, the representation of the user's hand or wrist is visible in the view of the three-dimensional environment.

Fig. 7A-7G illustrate displaying a user interface element on a view of a physical environment (or optionally a three-dimensional environment corresponding to and/or including a representation of one or more portions of the physical environment), wherein the user interface element has an appearance based on a graphical composition of content in the physical environment behind the user interface element at a plurality of different points in time, according to some embodiments. The user interfaces in fig. 7A-7G are used to illustrate the processes described below, including the process in fig. 8.

In some implementations, the user interface element presents a user interface such as an application, an operating system, or an alert. In some implementations, the user interface element is displayed as a three-dimensional object at a location in three-dimensional space. The user interface elements typically include a background based on the appearance of the underlying environment, as well as content with which the user may interact (e.g., while the background is typically non-interactive), such as text or graphics that may be scrolled, zoomed, and/or selected (e.g., configured), and/or icons or buttons that may be pressed or activated (e.g., configured). According to various embodiments, the behaviors described in fig. 7A-7G with respect to user interface elements in some examples apply to user interface elements in other examples unless otherwise indicated in the specification.

Fig. 7A-7Z illustrate an exemplary computer system (e.g., device 101 or another computer system) in communication with a display generation component (e.g., display generation component 7100 or another display generation component). As shown in the example in fig. 7A to 7Z, the display generating section 7100 is a touch screen held by the user 7002. In some embodiments, the display generating component of computer system 101 is a head mounted display worn on the head of the user (e.g., the content shown in fig. 7A-7Z as being visible via display generating component 7100 of computer system 101 corresponds to the field of view of user 7002 when the head mounted display is worn). In some embodiments, the display generating component is a stand-alone display, a projector, or another type of display. In some embodiments, the computer system communicates with one or more input devices including cameras or other sensors and input devices that detect movement of a user's hand, movement of the user's entire body, and/or movement of the user's head in a physical environment. In some embodiments, the one or more input devices detect movement and current pose, orientation, and position of a user's hands, face, and/or whole body. In some implementations, the user input is detected via a touch-sensitive surface or touch screen. In some embodiments, the one or more input devices include an eye tracking component that detects the location and movement of the user's gaze. In some embodiments, the display generating component and optionally the one or more input devices and computer system are part of a head mounted device (e.g., an HMD or a pair of goggles) that moves and rotates with the user's head in a physical environment and changes the user's point of view in a three-dimensional environment provided via the display generating component. In some embodiments, the display generating component is a heads-up display that does not move or rotate with the user's head or the entire body of the user, but optionally changes the user's point of view in a three-dimensional environment according to the movement of the user's head or body relative to the display generating component. In some embodiments, the display generating component (e.g., touch screen) is optionally moved and rotated by the user's hand relative to the physical environment or relative to the user's head, and the viewpoint of the user is changed in the three-dimensional environment according to the movement of the display generating component relative to the user's head or face or relative to the physical environment.

As shown in fig. 7B-7G, the computer system displays a view of the three-dimensional environment (e.g., environment 7104, virtual three-dimensional environment, augmented reality environment, a perspective view of the physical environment, or a camera view of the physical environment). In some embodiments, the three-dimensional environment is a virtual three-dimensional environment without a representation of a physical environment. In some embodiments, the three-dimensional environment is a mixed reality environment that is a virtual environment augmented by sensor data corresponding to a physical environment. In some embodiments, the three-dimensional environment is an augmented reality environment that includes one or more virtual objects and representations of at least a portion of the physical environment surrounding the display generating component 7100 (e.g., representations 7004 'and 7006' of walls, representation 7008 'of floors, and representation 7014' of physical objects 7014). In some implementations, the representation of the physical environment includes a camera view of the physical environment. In some embodiments, the representation of the physical environment includes a view of the physical environment through a transparent or translucent portion of the display generating component.

Fig. 7A illustrates an exemplary scene 105 in an exemplary physical environment 7102. In the scenario 105 in fig. 7A, the user 7002 is in a physical environment 7102, with a display 7100 of the computer system held in the user's hand 7022. Physical environment 7102 includes walls 7004 and 7006 and floor 7008. The physical box 7014 is positioned on the floor 7008. Further, the physical box 7012 falls toward the floor 7008 in space.

Fig. 7B illustrates a three-dimensional environment 7104 displayed via a display 7100. In the example shown in fig. 7B, the environment 7104 optionally includes a view of one or more portions of the scene 105 based on the portions of the scene 105 in the field of view of the computer system including the display 7100 (or more specifically, the field of view of one or more cameras of the computer system). For example, environment 7104 includes display walls 7004 'and 7006' that are representations of physical walls 7004 and 7006, respectively, display floor 7008 'that is a representation of physical floor 7008, and box 7014' that is a representation of physical box 7014. In the example shown in fig. 7B, the physical box 7012 is not in the field of view of the one or more cameras of the computer system, and thus no representation of the physical box 7012 is displayed in environment 7104 (e.g., as opposed to box 7012', which is a representation of the physical box 7012, shown in environment 7104, as shown in fig. 7C described herein). As shown in the example of fig. 7B, environment 7104 also optionally includes computer-generated objects (e.g., representations that do not correspond to physical objects), such as computer-generated wall-hanging 7106, which appears to be positioned on wall 7004 'like a physical object placed on wall 7004, and computer-generated balls 7108 and 7110 appear on floor 7008' as physical balls would appear on floor 7008. Ball 7110 is closer to the user's point of view than ball 7108, and both ball 7110 and ball 7108 are closer to the user's point of view than wall mount 7106. In fig. 7B, the view of the displayed environment 7104 corresponds to time t=t ₀.

Fig. 7C shows a view of the environment 7104 at a time t=t ₁ different from and later than the time t=t ₀ shown in fig. 7B. The environment 7104 as shown in fig. 7C reflects movement of the user's point of view down and back (e.g., away from) the wall 7004 in space (e.g., in response to movement of the computer system, or more specifically, movement of one or more cameras of the computer system down and back in space relative to the wall 7004). Thus, wall 7004 'in environment 7104 in fig. 7C appears farther from the user's point of view, and more walls 7006 'and floors 7008' are shown. In addition, the box 7014 'appears smaller in fig. 7C, farther from the user's point of view, and farther from the bottom edge of the display 7100 than in fig. 7B. Furthermore, wall-hanging 7106 is actually positioned on wall 7004 and thus is farther from the user's perspective in fig. 7C than in fig. 7B, shown smaller in environment 7104 in fig. 7C than in fig. 7B. Fig. 7C also shows that the bin 7012' is displayed in the environment 7104 according to the physical bin 7012 dropping such that the physical bin 7012 has entered the field of view of the computer system (or more specifically, the field of view of one or more cameras of the computer system). Further, in fig. 7C, the ball 7108 has scrolled to the right as indicated by the displaced position of the ball 7108 and by the indicia of the clockwise rotation on the ball 7108.

Fig. 7D shows a view of the environment 7104 at a time t=t ₂ later than the time T ₁ shown in fig. 7C and the time T ₀ shown in fig. 7B. Fig. 7D illustrates a transition from fig. 7C based on movement of the user's viewpoint via clockwise rotation (e.g., via clockwise rotation of the computer system). With the orientation of the environment 7104 remaining anchored to the physical environment 7102 in which the computer system is located (fig. 7A), the environment 7104 as shown in fig. 7D appears to be tilted (e.g., rotated counterclockwise) relative to the display 7100 of the computer system. The physical boxes 7012 continue to fall in space (e.g., in the physical environment 7102) such that more physical boxes 7012 have entered the field of view of the computer system, and thus more boxes 7012' representing physical boxes 7012 are displayed in the environment 7104. Further, the ball 7108 scrolls further to the right in the environment 7104 as indicated by the displaced position of the ball 7108 and the markings on the ball 7108 rotating further clockwise relative to fig. 7C.

Fig. 7D also shows a view of environment 7104 displayed in response to computer system 101 detecting an event that triggers display of user interface element 7030. In some embodiments, the user interface element 7030 is a user interface, such as an application user interface or a system user interface displayed within the environment 7104. In the example of fig. 7D, the user interface element 7030 is anchored to the viewpoint of the user instead of the environment 7104, as shown by the user interface element 7030 being tilted relative to the environment 7104 in fig. 7D (in contrast to the user interface element 7030 being upright relative to the environment 7104 in fig. 7E, as described in more detail below). The user interface element 7030 may be a user interface of an application executing on a computer system that is displayed in response to user input to launch the application. The user interface element 7030 can be an alert or notification displayed in response to an event occurring at the computer system (e.g., meeting one or more error conditions that display a reminder, such as reaching a predetermined time). Fig. 7D shows user interface element 7030 displayed over a portion of environment 7104, and in particular over a portion of box 7012', a portion of wall hanger 7106, a portion of ball 7108, and a portion of wall 7004'.

The appearance of at least a portion of the user interface element 7030 is based on the appearance of the underlying portion of the environment 7104 at one or more times prior to time t ₂ and/or at time t ₂. The appearance of generating user interface elements 7030 based on the underlying portions of environment 7104 is described in more detail herein with reference to fig. 7G.

Fig. 7E shows a view of the environment 7104 at a time t=t ₃ different from the time T ₂ shown in fig. 7D, the time T ₁ shown in fig. 7C, and the time T ₀ shown in fig. 7B. Fig. 7E shows a transition from fig. 7D. The environment 7104 as shown in fig. 7E reflects that the user's point of view is moved via a counter-clockwise rotation (e.g., to bring the computer system in place, and to reverse the clockwise rotation of the user's point of view shown in the transition from fig. 7C to fig. 7D) such that the environment 7104 appears horizontal relative to the display 7100 in fig. 7E. Furthermore, the physical boxes 7012 fall further spatially relative to the field of view of the computer system such that all corresponding boxes 7012' are behind and obscured by the user interface element 7030. Further, ball 7108 is shown scrolling further to the right in environment 7104 as indicated by the displaced position of ball 7108 and the further clockwise rotation of the marker on ball 7108 relative to environment 7104 shown in fig. 7D.

The user interface element 7030 in fig. 7E remains anchored to the point of view of the user (e.g., the perspective of the computer system) rather than the environment 7104. Thus, in response to the user's point of view being rotated counterclockwise to be substantially horizontal relative to the environment 7104 shown in fig. 7E, the user interface element 7030 is also rotated counterclockwise to be substantially upright relative to the environment 7104 (e.g., maintaining the orientation of the user interface element 7030 relative to the computer system). Those of ordinary skill in the art will readily appreciate that a user interface element such as user interface element 7030 may alternatively be anchored to environment 7104 such that the user interface element (e.g., user interface element 7030) does not move relative to environment 7104 despite the user's viewpoint changing relative to environment 7104. The appearance of the user interface element 7030 displayed above the environment 7104 in fig. 7E is different from the appearance of the user interface element 7030. Specifically, the appearance of the user interface element 7030 in fig. 7E is based on the appearance of the underlying portion of the environment 7104 at a different time than the time used for the appearance of the user interface element 7030 in fig. 7D. For example, the appearance of the user interface element 7030 in fig. 7E is based on the appearance of the underlying portion of the environment 7104 at one or more times prior to time t ₃ and/or at time t ₃, as described in more detail herein with reference to fig. 7G.

Fig. 7F illustrates different simulated positions of the user interface element 7030 relative to the environment 7104, as shown in a top view 7034 of the environment 7104. Top view 7034 shows the position of the user's viewpoint 7032 in the environment 7104, the positions of the balls 7110 and 7108, and several different positions 7030-1 to 7030-4 of the user interface element 7030 in a three-dimensional environment. The different simulated positions shown in fig. 7F are possible simulated positions of the user interface element 7030 shown in fig. 7D and 7E in the environment 7104. For example, the user interface element 7030 in fig. 7D and/or 7E may be in the environment 7104 at a simulated location corresponding to location 7030-1 in fig. 7F, which is in front of other objects shown in the environment 7104 (such as ball 7110 and box 7012 ') from the user's point of view 7032.

In another example, the user interface element 7030 in fig. 7D and/or 7E may be at a simulated position in environment 7104 corresponding to position 7030-2 in fig. 7F, wherein position 7030-2 is farther from viewpoint 7032 than position 7030-1 (e.g., the distance between position 7030-2 and viewpoint 7032 is greater than the distance between position 7030-1 and viewpoint 7032), farther from viewpoint 7032 than case 7012' (e.g., the simulated position of case 7012' in environment 7104 corresponds to the physical position of case 7012 in physical environment 7102), farther from viewpoint 7032 than ball 7108 (e.g., the simulated position of the ball), and closer to viewpoint 7032 than ball 7108 (e.g., the simulated position of the ball) (and other objects behind ball 7108, such as wall-mount 7106 and wall-mount 7004 ').

In another example, user interface element 7030 in fig. 7D and/or 7E may be at a simulated position in environment 7104 corresponding to position 7030-3 in fig. 7F, wherein position 7030-3 is farther from viewpoint 7032 than positions 7030-1 and 7030-2 (e.g., the distance between position 7030-3 and viewpoint 7032 is greater than the distance between position 7030-2 and viewpoint 7032), farther from viewpoint 7032 than ball 7108 (e.g., the simulated position), and closer to viewpoint 7032 than wall-hanging 7106 and wall 7004.

In yet another example, the user interface element 7030 in fig. 7D and/or 7E can be at a simulated position in the environment 7104 that corresponds to position 7030-4 in fig. 7F, wherein position 7030-4 is farther from the viewpoint 7030 than positions 7032-1 through 7030-3 (e.g., the distance between position 7030-4 and viewpoint 7032 is greater than the distance between position 7030-3 and viewpoint 7032), and even farther from the viewpoint 7032 than wall 7004'.

As described in greater detail herein with reference to fig. 7G, in some embodiments, the appearance of the user interface element 7030 is based on the portion of the environment 7104 that overlaps the user interface element 7030, whether the overlapping portion is behind the simulated position of the user interface element 7030 or in front of the simulated position of the user interface element 7030. In some cases, the appearance of the user interface element 7030 is based on the portion of the environment 7104 that follows the user interface element 7030. In some cases, the appearance of the user interface element 7030 is based on a portion of the environment 7104 that is in front of the simulated position of the user interface element 7030 (e.g., even though the user interface element 7030 is displayed overlaid on an overlapping portion of the environment 7104 and thereby obscures that portion).

Fig. 7G illustrates a manner in which the appearance of a user interface element (e.g., user interface element 7030) is generated based on the graphical composition of the appearance of the overlapping portion of the three-dimensional environment (e.g., environment 7104) covered by the user interface element. In the scenario 7036 in FIG. 7G, the user interface elements 7030-D have the same appearance as the user interface element 7030 in FIG. 7D. Scene 7036 shows that the appearance of user interface elements 7030-D at time t ₂ (e.g., the appearance of user interface element 7030 as shown in fig. 7D, which shows a view of environment 7104 at time t ₂) is a composition based on a plurality of frames generated at time t ₂, wherein different frames correspond to a portion of the view of environment 7104 at different times. In some embodiments, when basing the appearance of a respective user interface element on a composition, as described herein with reference to fig. 7G, one or more visual transformations are applied to one or more regions of the composition to generate a resulting appearance of the user interface element, as described herein in more detail with reference to fig. 7X-7Z. The composition in scene 7036 is based on the appearance of a portion of the first view of environment 7104 at time t ₀ (fig. 7B), which corresponds to the portion of environment 7104 on which user interface element 7030 in fig. 7D is displayed, indicated by the dense dashed line; a portion of the second view of the environment 7104 at time t ₁ (fig. 7C), which corresponds to a portion of the environment 7104 on which the user interface element 7030 in fig. 7D is displayed, indicated by a dash-dot line; and a portion of the third view of environment 7104 at time t ₂ on which user interface element 7030 in fig. 7D is displayed, indicated by a dashed line. In the example of scene 7036, the central portion of the composition indicated by the solid line defines the appearance of user interface elements 7030-d.

In some embodiments, as shown in FIG. 7D, the user interface element 7030 includes a background portion 7030-a, a first content portion 7030-b, and a second content portion 7030-c. The content portions 7030-a and 7030-b optionally include visual content such as text or graphics, or include selectable and/or activatable objects such as buttons or controls. In some embodiments, the appearance of the background portion of the user interface element (e.g., background portion 7030-a of user interface element 7030) is based on a composition of the appearance of the underlying three-dimensional environment at a different prior time. In some embodiments, the appearance of the content portion (e.g., content portion 7030-b and/or 7030-c) of the user interface element is based on the content displayed in the content portion, and optionally is not based on the composition (e.g., although the content portion may be transparent or partially transparent such that at least some of the background portion having an appearance based on the composition is visible, as shown in fig. 7G).

In some cases, the composition is based on the appearance of a portion of a view of the three-dimensional environment when no user interface element is displayed. For example, the appearance of the user interface element 7030 in fig. 7D is based on the appearance of the corresponding portion of the environment 7104 in fig. 7B and 7C, even though the user interface element 7030 was not displayed at time t ₀ and time t ₁ (e.g., because the user interface element 7030 was displayed in response to an event occurring at the computer system after time t ₁ and before time t ₂). In some embodiments, the portion of the view of environment 7104 that is incorporated into the composition corresponds to the area of display 7100 that is occupied by user interface element 7030. For example, fig. 7D illustrates user interface elements 7030 occupying respective areas of the display 7100. Although the user interface element 7030 is not shown in fig. 7C, in some embodiments, the portion of the view of the environment 7104 that occupies the corresponding area of the display 7100 in fig. 7C is incorporated into the composition. In some embodiments, the portion of the view of environment 7104 that is incorporated into the composition includes all or substantially all (e.g., 90% or greater) of the portion that is displayed in the corresponding region of display 7100 that is occupied (or later occupied) by user interface element 7030. In some embodiments, the portion of the view of environment 7104 that is incorporated into the composition extends beyond the corresponding area of display 7100 that is occupied (or later occupied) by user interface element 7030 (e.g., frames t ₀、t₁ and t ₂ extend beyond the solid outline of the central portion of the composition, which optionally delineates user interface element 7030-d).

In the example in scene 7036, the portion of the view of environment 7104 in fig. 7B that is included in the frame and incorporated into the composition from time t ₀ is the portion that includes a portion of the left side of wall-mount 7106 in the left portion of the frame and a portion of the top of ball 7108 in the bottom portion of the frame. Frame t ₀ does not include any portion of box 7012' that was not displayed in the view of environment 7104 at time t ₀ (e.g., because physical box 7012 has not entered the field of view of one or more cameras corresponding to the displayed portion of environment 7104).

The portion of the view of environment 7104 in fig. 7C that is included in the frame starting from time t ₁ and that is incorporated into the composition in scene 7036 is different from the portion of frame t ₀, depending on the viewpoint of the user and the environment 7104 changes. Specifically, frame t ₁ includes a portion of wall hanging 7106 to the left in the left portion of the frame that is located higher than the portion of wall hanging 7106 in frame t ₀, and is less proportional (e.g., because the user's point of view in fig. 7C has moved downward and is farther from wall 7004' and wall hanging 7106 than the user's point of view in fig. 7B). Frame t ₁ also includes a portion of the top of ball 7108 in the bottom portion of the frame, which is a larger portion of the top of ball 7108 than the portion included in frame t ₀. In particular, since the viewpoint of the user in FIG. 7C has moved downward and is farther from wall 7004' than the viewpoint of the user in FIG. 7B, this portion of the FIG. 7C view of environment 7104 in frame t ₁ includes more of the ball 7108. As ball 7108 scrolls further to the right in fig. 7C relative to fig. 7B, ball 7108 is also more to the right in frame t ₁ than in frame t ₀. In the top portion of the frame, frame t ₁ also includes a portion of the bottom of box 7012', which is displayed in the view of environment 7104 in fig. 7C (e.g., according to the fact that box 7012 falls in space and into the field of view of the computer system corresponding to that portion of the view of environment 7104 in frame t ₁).

According to the clockwise rotation of the user's viewpoint to the viewpoint shown in fig. 7D, the portion of the view of environment 7104 that is included in the frame starting from time t ₂ and incorporated into scene 7036 reflects the change in environment 7104 and is tilted with respect to the portion of the view of environment 7104 in frames t ₀ and t ₁. Specifically, frame t ₂ includes more of the left side of wall hanging 7106 on the left side of the frame than frame t ₁, and the included portion of wall hanging 7106 is tilted counterclockwise. Since the ball 7108 has scrolled further to the right, the frame t ₂ also includes a smaller portion of the top of the ball 7108 at the bottom portion of the frame, and the included portion of the ball 7108 is also tilted counterclockwise (e.g., to a discernable extent, given that the ball 7108 is circular). Further, frame t ₂ includes more of the box 7012' in the top portion of the frame (e.g., according to the fact that box 7012 falls in space such that more of the box 7012 is in the field of view of the computer system corresponding to the displayed view of environment 7104 and correspondingly such that more of the box 7012' is in the portion of environment 7104 on which user interface element 7030 in fig. 7D is displayed), and the included portion of box 7012' is also tilted counterclockwise relative to frame t ₂.

In some embodiments, as shown in fig. 7G, the frames used in the composition may be offset from one another based on the viewpoint of the user at the time associated with each frame. For example, according to the viewpoint of the user in fig. 7B (for frame t ₀) being different from the viewpoint of the user in fig. 7C (for frame t ₁), the viewpoint of the user has moved down from fig. 7B to fig. 7C and is farther away from wall 7004', so that when the composition is formed, frame t ₀ is positioned to the right and/or scaled up (e.g., pushed closer) and down than frame t ₁. In other words, the offset and/or scaling of frame t ₁ relative to frame t ₀ is consistent with the change in user viewpoint such that the content of frame t ₁ is moved up and out of distance relative to the content of frame t ₀. Similarly, according to the user's viewpoint rotating clockwise in fig. 7D with respect to fig. 7B to 7C, the content of the frame t ₂ is tilted counterclockwise with respect to the content of the frames t ₀ and t ₁.

In some embodiments, the appearance of the user interface element 7030 is based on a combination of the appearances of the portions of the environment 7104 even though the simulated locations of the user interface element 7030 are farther from the user's point of view than those portions of the environment 7104, as discussed above with reference to fig. 7F. For example, the simulated position of the user interface element 7030 in fig. 7D may be position 7030-4 as shown in fig. 7F. In this example, the appearance of user interface element 7030-D is based on the appearance of frames t ₀、t₁ and t ₂, which include a view of the object portions (e.g., wall-hanging 7106, ball 7108, and box 7012 'in frames t ₁ and ₂) that are closer to the user's viewpoint 7032 in environment 7104 than to location 7030-4, even though user interface element 7030 in fig. 7D covers or replaces the view of those portions of the closer objects. In another example, the simulated position of user interface element 7030 in fig. 7D is position 7030-3 as shown in fig. 7F, and the appearance of user interface element 7030-D is based on the appearance of portions of environment 7104 that include portions of wall-hanging 7106, ball 7108, and box 7012', even though position 7030-3 is farther from the user's point of view 7032 than ball 7108 and box 7012 '. In yet another example, the simulated position of the user interface element 7030 in fig. 7D is position 7030-2 and the appearance of the user interface element 7030-D is also based at least in part on the appearance of the box 7012', even though the box 7012' is closer to the viewpoint 7032 than the position 7030-2. Finally, with respect to FIG. 7F, user interface element 7030 may have a simulated position of position 7030-1 that is closer to the user's point of view 7032 than any of boxes 7012', balls 7108, and wall hanging 7106.

In some embodiments, generating the composition includes assigning different weights to different frames. Optionally, frames that are given greater weight in generating the composition have a greater impact on the appearance of the user interface element 7030 than frames that are given less weight. In the context 7036, the appearance of the user interface elements 7030-d can be based on a first amount (e.g., a first degree or range) of frame t ₀, a second, different amount (e.g., a second degree or range) of frame t ₁, and a third, different amount (e.g., third degree or range). For example, if frame t ₂ is weighted higher than frame t ₁, and frame t ₁ is weighted higher than frame t ₀, The top portion of the user interface element 7030-d (e.g., delineated by a solid line in the composition) is based on the appearance of the bottom portion of the box 7012' in frame t ₂ (e.g., wherein the box 7012' spans the solid outline) to a greater extent than the appearance of the corresponding portion of frame t ₁ (e.g., wherein the box 7012' is outside the solid outline), and even greater than the appearance of the corresponding portion of frame t ₀ (e.g., wherein the box 7012' is not shown). In other words, the box 7012' contributes significantly more to the appearance of the user interface elements 7030-d as shown in the scene 7036 than if the weight of the frame t ₂ was lower. Continuing with the same example, the lower right portion of user interface element 7030-d is based more on the appearance of the top portion of ball 7108 in frame t ₂ than the appearance of the corresponding portions of frame t ₁ and frame t ₀. In other words, if the weight of frame t ₁ is higher, then the contribution of ball 7108 to the appearance of user interface element 7030-d as shown in scene 7036 will be more pronounced because ball 7108 in frame t ₁ extends farther into the solid outline than in frames t ₀ and t ₂.

In some embodiments, the weighting of frames in the composition is based on whether the time of these frames is close or far from the time of formation of the composition. For example, in some embodiments, the contribution of frames closer in time to the composition being generated (e.g., later, newer frames) is greater than the contribution of frames farther in time from the composition being generated (e.g., earlier, less newer frames). Using the example in scenario 7036, according to time t ₁ being closer to time t ₂ than time t ₀ to time t ₂, the composition generated at time t ₂ optionally includes a greater contribution from frame t ₁ than from frame t ₀, such as a greater contribution from bin 7012 'in frame t ₁ than the contribution from the absence of bin 7012' in frame t ₀.

In some embodiments, one or more transformations or modifications are applied when generating the composition (e.g., first applied to the frame before the frame is used to generate the composition, or applied to an intermediate composition of unmodified frames to generate the final composition). In some cases, the transformation/alternation serves to smooth out the appearance of the frame and/or the resulting composition, which improves the legibility of the content displayed overlaying the user interface element. For example, the composition and/or the frames used to generate the composition may be blurred. In another example, the resolution of the image or graphical information in the frame and/or resulting composition may be reduced (e.g., by downsampling). In another example, a noise pattern may be applied to the composition and/or frames used to generate the composition. Thus, scene 7036 in fig. 7G illustrates a user interface element 7030-d having a blurred appearance achieved by blurring frames t ₀、t₁ and/or t ₂ and/or applying a noise pattern when generating a composition at time t ₂. In some embodiments, the frames are made partially translucent for use in generating the composition and/or the resulting composition is made partially translucent, which will allow the underlying environment 7104 to be at least partially visible under the user interface element 7030.

Returning to fig. 7G, scene 7038 illustrates a manner of generating an appearance of user interface element 7030-E that has the same appearance as user interface element 7030 in fig. 7E. Scene 7038 shows that the appearance of user interface element 7030 at time t ₃ (e.g., the appearance of user interface element 7030 as shown in fig. 7E, which shows a view of environment 7104 at time t ₃) is a composition based on a plurality of frames generated at time t ₃, Wherein different frames correspond to portions of the view of the environment 7104 at different times. The appearance of the user interface element 7030 at time t ₃ (fig. 7E) is optionally based on a different frame (fig. 7D) than the frame used to generate the composition that serves as the basis for the appearance of the user interface element 7030 at time t ₂. Specifically, the composition in scene 7038 is based on the appearance of a portion of the first view of environment 7104 at time t ₁ (fig. 7C), which corresponds to the portion of environment 7104 on which user interface element 7030 in fig. 7E is displayed, indicated by the dash-dot line; The environment 7104 displays a portion of the second view of time t ₂ (fig. 7D) with the user interface element 7030 thereon (or alternatively, a portion of the view of environment 7104 at time t ₂, which corresponds to a portion of environment 7104 later having the user interface element 7030 displayed thereon at time t ₃, as shown in fig. 7E), indicated by the dashed line; And a portion of the third view of environment 7104 at time t ₃ on which user interface element 7030 in fig. 7E is displayed, indicated by the sparse dashed line. In the example of scene 7038, the central portion of the composition indicated by the solid line defines the appearance of user interface element 7030-e. In the example shown in fig. 7G, the composition in scene 7038 differs from the composition in scene 7036 in that the composition in scene 7038 includes frame t ₃ and does not include frame t ₀. One of ordinary skill in the art will recognize that different combinations of current and/or previous frames may be included (or excluded) in the composition defining the appearance of the user interface element at a given point in time relative to frames of the composition used to determine the appearance of the user interface element at other points in time.

As discussed above with reference to scene 7036, the composition may in some cases be based on the appearance of a portion of the view of environment 7104 when user interface element 7030 is not displayed. Scene 7038 shows the appearance of a portion of the view of the composition at time t ₁ based on environment 7104 shown in fig. 7C, prior to displaying user interface element 7030. In this case, the portion of the view of environment 7104 that is incorporated into the composition corresponds to the area of display 7100 that is occupied by user interface element 7030 at a later time, such as t ₂ or t ₃. Furthermore, in some embodiments, the portion of the view of environment 7104 that is included in a particular frame incorporated into the composition extends beyond the boundary of user interface element 7030 (e.g., frames t ₁、t₂ and t ₃ extend beyond the solid outline of the central portion of the composition, which optionally delineates user interface elements 7030-e).

Frames t ₁ and t ₂ used in the composition in scene 7038 are taken from the view of environment 7104 described with reference to fig. 7C and 7D, respectively, which are identical to those used in the composition described above with reference to scene 7036. According to the counterclockwise rotation of the user's viewpoint in the transition from fig. 7D to fig. 7E, the portion of the view of environment 7104 in fig. 7E that is included in the frame starting from time t ₃ and that is incorporated into the composition in scene 7038 (e.g., instead of frame t ₀) is rotated clockwise relative to the portion of frame t ₂ such that frame t ₃ comprises a horizontal view of environment 7104. Specifically, frame t ₃ includes a portion of the left side of wall mount 7106 in the left portion of the frame and a portion of the top of ball 7108 in the bottom portion of the frame. Frame t ₃ also includes more of the box 7012 'in the top portion of the frame (e.g., according to the physical box 7012 falling in space such that more of the box 7012' is in the portion of environment 7104 on which the user interface element 7030 in fig. 7E is displayed).

Thus, the user interface element 7030-e in the scene 7038 has a different appearance than the user interface element 7030-d in the scene 7036 because the user interface element 7030-e is based on a composition of different frames having different content than the frames used for the composition on which the user interface element 7030-d is based. For example, because the box 7012' extends to the extent of frame t ₃, and because the user interface elements 7030-e are based on a composition that includes frame t ₃, the contribution of the box 7012' to the appearance of the user interface elements 7030-e as shown in scene 7038 is more pronounced than the contribution of the box 7012' to the appearance of the user interface elements 7030-d as shown in scene 7036.

As discussed above with reference to scene 7036, the frames used in the composition may be offset from one another based on the viewpoint of the user at the time associated with each frame. For example, in scene 7038, the content of frame t ₂ rotates (e.g., counterclockwise) relative to the content of frames t ₁ and t ₃ according to the user's viewpoint rotating (e.g., clockwise) relative to the user's viewpoint in fig. 7C and 7E.

As also discussed above with reference to scene 7036, the appearance of user interface element 7030 is optionally based on a combination of the appearances of portions of environment 7104, even though the simulated locations of user interface element 7030 are farther from the user's viewpoint than those portions of environment 7104. For example, the simulated positions of the user interface element in fig. 7E may be any of the positions 7030-1 through 7030-4, and the user interface element 7030-E may have the same appearance for any of those simulated positions.

Further, as discussed above with reference to scene 7036, in some embodiments, one or more transformations or modifications are applied in scene 7038 when the composition is generated at time t ₃, such as blurring, applying a noise pattern, and/or downsampling image or graphics information. Thus, the scene 7038 shows a user interface element 7030-e with a blurred appearance achieved by blurring, downsampling, and/or applying a noise pattern to the frames t ₁、t₂ and/or t ₃ when the composition is generated at time t ₃.

Similarly, in scenario 7038, generating a composition optionally includes assigning different weights to different frames. For example, the appearance of the user interface element 7030-e may be based on a first amount of frame t ₁, a second, different amount of frame t ₂, and a third, different amount of frame t ₃. For example, if frame t ₃ is weighted higher than frame t ₂, and frame t ₂ is weighted higher than frame t ₁, The top portion of the user interface element 7030-e (e.g., delineated by a solid line in the composition) is more based on the appearance of the bottom portion of the box 7012' in frame t ₃ (e.g., wherein the box 7012' extends farther into the frame) than the corresponding portion of frame t ₂ (e.g., wherein the box 7012' extends less into the frame and also rotates counterclockwise relative to the frame). In other words, the box 7012' contributes significantly more to the appearance of the user interface element 7030-e as shown in the scene 7038 than if the weight of the frame t ₃ was lower. Continuing with the same example, the right portion of user interface element 7030-e is based more on the appearance of the left portion of wall-hanging 7106 in frame t ₃ than the appearance of the corresponding portions of frames t ₂ and t ₁. In other words, if frame t ₂ is weighted higher, wall hanging 7106 will contribute more significantly to the appearance of user interface element 7030-e as shown in scene 7038, because wall hanging 7106 in frame t ₂ extends farther into the solid outline than in frames t ₁ and t ₃.

As another example of assigning a greater weight to a frame from a time closer to the time at which the composition was generated (e.g., a later, newer frame) than to a frame from a time farther from the time at which the composition was generated (e.g., an earlier, less newer frame), in scene 7038, the weight of frame t ₃ is higher than frame t ₁, which is consistent with generating the composition at time t ₃ (which occurs after time t ₁). Accordingly, in the composition generated at time t ₃, the contribution from frame t ₃ is greater than the contribution from frame t ₁, e.g., the contribution from bin 7012 'in frame t ₃ is greater than the contribution from the smaller portion of bin 7012' in frame t ₁.

In a comparison of the composition generated at time t ₂ (scenario 7036) with the composition generated at time t ₃ (scenario 7038), it is also evident that the contribution of the later frames to the composition is greater. For example, both compositions include frame t ₁. However, time t ₁ is closer to time t ₂ than time t ₃, so the composition generated at time t ₂ (scene 7036) includes a greater contribution from frame t ₁ than the composition generated at time t ₃. In particular, the method comprises the steps of, The contribution of the ball 7108 in frame t ₁ to the composition generated at time t ₂ (scene 7036) is greater than the contribution of the ball 7108 in frame t ₁ to the composition generated at time t ₃ (scene 7038), So that the effect of the ball 7108 is more pronounced in the user interface element 7030-d than in the user interface element 7030-e. In another example, the absence of bin 7012' in frame t ₀ contributes to the composition generated at time t ₂ (scenario 7036), while frame t ₀ is not included in and does not contribute to the composition generated at time t ₃ (scenario 7038), And alternatively, the composition generated at time t ₃ is based on an additional frame t ₃ that does include a portion of the box 7012', such that the top portion of the resulting user interface element 7030-d is less affected by the presence of the box 7012' than the top portion of the user interface element 7030-e.

In some embodiments, the number of frames included in the composition is based on the amount of movement of the user's viewpoint. For example, in fig. 7G, the composition generated at time t ₂ includes three frames t ₀、t₁ and t ₂ based on the movement amount of the viewpoint of the user between fig. 7B to 7D. Similarly, the composition generated at time t ₃ includes three frames t ₁、t₂ and t ₃ based on the amount of movement of the user's viewpoint between fig. 7C to 7E. One of ordinary skill in the art will recognize that with less movement of the user's viewpoint, fewer frames may be included when generating the composition. For example, if the view of environment 7104 remains the same in fig. 7D as in fig. 7C, the composition generated at t ₂ may include two frames instead of three frames (e.g., frame t ₀ and only one other frame selected from frames t ₁ and t ₂). In other words, as the user's viewpoint moves more, the appearance of the underlying environment is sampled more frequently, such that the time interval between different frames included in the composition is larger when the user's viewpoint moves less, and smaller when the amount of movement of the viewpoint for the user is more.

Fig. 7H-7J illustrate displaying a user interface element on a view of a three-dimensional environment, where the user interface element has an appearance based on a color gradient generated from one or more colors sampled from a plurality of points in the three-dimensional environment, according to some embodiments. The user interfaces in fig. 7H-7J are used to illustrate the processes described below, including the process in fig. 9.

In some embodiments, as described herein with reference to fig. 7A-7G, the user interface element presents a user interface and/or is displayed as a three-dimensional object at a location in three-dimensional space. The user interface element typically includes a background based on a color gradient generated from one or more colors sampled from the underlying environment and content with which a user may interact. According to various embodiments, the behaviors described in fig. 7H-7J with respect to user interface elements in some examples apply to user interface elements in other examples unless otherwise indicated in the specification. In some embodiments, when basing the appearance of the respective user interface element on a color gradient, as described herein with reference to fig. 7H-7J, one or more visual transforms are applied to one or more regions of the color gradient to generate a resulting appearance of the user interface element, as described herein in more detail with reference to fig. 7X-7Z.

Fig. 7H shows a view of environment 7104 at time t=t ₂ (e.g., fig. 7H is an alternative to the scene shown in fig. 7D). In fig. 7H, display generation component 7100 displays a view of environment 7104 in response to computer system 101 detecting an event that triggers the display of user interface element 7040 instead of user interface element 7030. Similar to the user interface element 7030 (fig. 7D), the user interface element 7040 can be an application user interface (e.g., displayed in response to user input to launch an application), an alert, a notification, an operating system user interface, or other computer-generated object. In contrast to fig. 7D, in this figure, user interface element 7030 is displayed with an appearance based on the graphical composition of the underlying portion of environment 7104 at a different time, including time before time t ₂ (and optionally including t ₂), but in fig. 7H, user interface element 7040 is displayed with an appearance based on the color gradient generated from the color sampled by environment 7104. In the example shown in fig. 7H, a color gradient is generated from the color sampled from environment 7104 at a time corresponding to time t ₂ (e.g., at time t ₂, at a time prior to t ₂ that results in the gradient being displayed at time t ₂ shown in fig. 7H after generation, or during a time range that includes or ends at t ₂). In some embodiments, the color gradient generated at a given time or time range is less computationally intensive than the graphics composition generated from multiple frames at different points in time.

Specifically, a color gradient is generated from colors sampled from the portion of the view of the environment 7104 (represented as region 7042 in fig. 7H (left)) covered by the user interface element 7040. That is, the appearance of the user interface element 7040 is based on a color gradient between multiple colors (e.g., two or more colors) sampled from the region 7042 of the environment 7104. In the example of fig. 7H, the sampled colors include a first color 7044-1 taken from box 7012', a second color 7044-2 taken from wall hanging 7106, and a third color 7044-3 taken from wall 7004'. Thus, the user interface element 7040 is displayed in FIG. 7H (right) in a color gradient between colors 7044-1, 7044-2, and 7044-3. Specifically, the color gradient from the upper left corner of user interface element 7040 overlaid on box 7012 'to the upper right corner of user interface element 7040 overlaid on wall-hanging 7106 transitions from color 7044-1 sampled according to box 7012' to color 7044-2 sampled according to box 7106. Similarly, the color gradient from the upper left corner of the user interface element 7040 overlaid on the box 7012 'to the lower left corner of the user interface element 7040 overlaid on the wall 7004' transitions from the color 7044-1 sampled according to the box 7012 'to the color 7044-3 sampled according to the wall 7004'. Likewise, the color gradient from the upper right hand corner of user interface element 7040 overlaid on wall hanging 7106 to the lower right hand corner of user interface element 7040 overlaid on wall 7004 'transitions from color 7044-2 sampled according to box 7106 to color 7044-3 sampled according to wall 7004'. The gradient of color along the bottom edge region of the user interface element 7040 is based on two colors both sampled from the wall 7004', which are the same color in the example of fig. 7H, and thus the bottom edge region of the user interface element 7040 is shown without a discernible gradient.

In some implementations, as shown in FIG. 7H, the user interface element 7040 includes a background portion 7040-a, a first content portion 7040-b, and a second content portion 7040-c. The content portions 7040-a and 7040-b optionally include visual content such as text or graphics, or include selectable and/or activatable objects such as buttons or controls. In some implementations, the appearance of the background portion of the user interface element (e.g., background portion 7040-a of user interface element 7040) is based on a color gradient generated from colors sampled from different portions of the underlying three-dimensional environment. In some implementations, the appearance of the content portions (e.g., content portions 7040-b and/or 7040-c) of the user interface element is based on content displayed in the content portions, and optionally is not based on color gradients (e.g., although the content portions may be transparent or partially transparent such that at least some of the background portions having an appearance based on color gradients are visible, as shown in fig. 7H).

In some embodiments, as shown in fig. 7H-7J, a user interface element, such as user interface element 7040, is opaque. That is, the user interface element 7040 (specifically, including the background portion 7040-a) is displayed with an appearance based on the generated color gradient and is not displayed with a transparency that allows the underlying three-dimensional environment to be at least partially visible through the user interface element 7040. In some embodiments, as shown in fig. 7H-7J, the effect of the color gradient on the appearance of the user interface element 7040 extends to the edge of the user interface element 7040 (specifically, including the background portion 7040-a).

Fig. 7I shows a view of environment 7104 at a time t=t ₃ different from time T ₂ shown in fig. 7H (e.g., fig. 7I is an alternative to the scene shown in fig. 7E). Fig. 7I shows an exemplary transition from fig. 7H. The environment 7104 as shown in fig. 7I reflects that the user's viewpoint is moved via a counterclockwise rotation (e.g., similar to the change of the user's viewpoint from fig. 7D to fig. 7E) such that the environment 7104 appears horizontal with respect to the display 7100 in fig. 7I. The user interface element 7040 remains anchored to the point of view of the user (e.g., the perspective of the computer system) rather than the environment 7104. Thus, in response to the user's point of view being rotated counterclockwise to be substantially horizontal relative to the environment 7104, the user interface element 7040 is also rotated counterclockwise to be substantially upright relative to the environment 7104 (e.g., maintaining the orientation of the user interface element 7040 relative to the computer system). In fig. 7I, the appearance of the user interface element 7040 displayed over the environment 7104 is based on a color gradient generated from colors sampled from the environment 7104 at a time or time range corresponding to time t ₃ (e.g., at time t ₃, at a time prior to time t ₃ resulting in a gradient being displayed at time t ₃ as shown in fig. 7I, or during a time range that includes or ends at t ₃), and in particular, based on the portion of the environment 7104 on which the user interface element 7040 is displayed.

Specifically, a color gradient is generated from the colors sampled from the region 7046 in fig. 7I (left), which indicates the portion of the view of the environment 7104 covered by the user interface element 7040. In the example of fig. 7I, the sampled colors include a first color 7044-4 (e.g., different from color 7044-1 in fig. 7H) taken from box 7012', a second color 7044-5 (e.g., different from color 7044-2 in fig. 7H) taken from wall-mount 7106, and a third color 7044-6 (e.g., different from color 7044-3 in fig. 7H) taken from wall 7004'. Thus, the user interface element 7040 is displayed in FIG. 7I (right) in a color gradient between colors 7044-4, 7044-5, and 7044-6. Specifically, the color gradient from the upper left corner of user interface element 7040 overlaid on box 7012 'to the upper right corner of user interface element 7040 overlaid on wall-hanging 7106 transitions from color 7044-4 sampled according to box 7012' to color 7044-5 sampled according to box 7106. Similarly, the color gradient from the upper left corner of the user interface element 7040 overlaid on the box 7012 'to the lower left corner of the user interface element 7040 overlaid on the wall 7004' transitions from the color 7044-4 sampled according to the box 7012 'to the color 7044-6 sampled according to the wall 7004'. Likewise, the color gradient from the upper right hand corner of user interface element 7040 overlaid on wall hanging 7106 to the lower right hand corner of user interface element 7040 overlaid on wall 7004 'transitions from color 7044-5 sampled according to box 7106 to color 7044-6 sampled according to wall 7004'. The gradient of color along the bottom edge region of the user interface element 7040 is based on two colors both sampled from the wall 7004', which are the same color in the example of fig. 7I, and thus the bottom edge region of the user interface element 7040 is shown without a discernible gradient.

In some cases, the color of the portion of the environment 7104 behind the user interface element 7040, and thus the sampled color, is the same from one point in time to another, such as if the user's point of view is not changed from fig. 7H to fig. 7I, and the portion of the environment 7104 behind the user interface element 7040 in fig. 7I is the same as the portion in fig. 7H (e.g., if color 7044-4 is the same as color 7044-1, if color 7044-5 is the same as color 7044-2, and if color 7044-6 is the same as color 7044-3). In this case, the color gradient aspect of the appearance of the user interface element 7040 is the same in fig. 7I as in fig. 7H. In some embodiments, after generating the color gradient of the appearance of the user interface element 7040 at time t ₂, the computer system will not need to regenerate the color gradient of the appearance of the user interface element 7040 at time t ₃, which will reduce the computational burden of generating the appearance of the user interface element 7040 and/or displaying the user interface element.

Fig. 7J shows a view of the environment 7104 at a time t=t ₄ different from the time T ₃ shown in fig. 7I. Fig. 7J shows an exemplary transition from fig. 7I. The environment 7104 as shown in fig. 7J reflects movement of the user interface element 7040 rightward and slightly downward from its position as shown in fig. 7I. The appearance of the user interface element 7040 in fig. 7J is based on a color gradient generated from the color sampled from the environment 7104 at a time or time range corresponding to time t ₄, and in particular, on the portion of the environment 7104 on which the user interface element 7040 is displayed.

The region 7048 in fig. 7J (left) indicates a portion of the view of the environment 7104 covered by the user interface element 7040. In fig. 7J (right), user interface element 7040 is overlaid on a portion of environment 7104 that includes wall hanging 7106 and box 7014'. The appearance of the user interface element 7040 is accordingly based on a color gradient between the plurality of colors sampled from the region 7048 of the environment 7104. In the example of FIG. 7H, the colors sampled include a first color 7044-7 taken from wall hanging 7106, a second color 7044-8 taken from wall 7004', a third color 7044-9 taken from box 7014', and a fourth color 7044-10 taken from wall 7004 '. Thus, in FIG. 7J (right), the user interface element 7040 is displayed with a color gradient between colors 7044-7, 7044-8, 7044-9, and 7044-10. In particular, the color gradient from the upper left corner of user interface element 7040 overlaid on wall hanging 7106 to the upper right corner of user interface element 7040 overlaid on wall 7004 'transitions from color 7044-7 sampled according to wall 7106 to color 7044-8 sampled according to wall 7004'. Similarly, the color gradient from the upper left corner of user interface element 7040 overlaid on wall hanging 7106 to the lower left corner of user interface element 7040 overlaid on wall 7004 'transitions from color 7044-7 sampled according to wall 7106 to color 7044-10 sampled according to wall 7004'. Likewise, the color gradient from the upper right hand corner of the user interface element 7040 overlaid on the wall 7004 'to the lower right hand corner of the user interface element 7040 overlaid on the box 7014' transitions from the color 7044-8 sampled according to the wall 7004 'to the color 7044-9 sampled according to the box 7014'. Further, the color gradient along the bottom edge region of the user interface element 7040 transitions from the color 7044-10 sampled according to the wall 7004 'to the color 7044-9 sampled according to the box 7014' from the lower left corner of the user interface element 7040 overlaid on the wall 7004 'to the lower right corner of the user interface element 7040 overlaid on the box 7014'.

In some embodiments, the appearance of the user interface element at a given time (such as the appearance of the user interface element 7040 at time t ₂) is based on using the color sampled at the time corresponding to the given time rather than using the color gradient of the color sampled at the time corresponding to the time prior to the given time, such as time t ₁ or time t ₀, thereby reducing the computational burden of generating the color gradient and thus the user interface element. In another example, the appearance of the user interface element 7040 at time t ₄ is based on using the color sampled at a time corresponding to time t ₄ instead of using the color gradient of the color sampled at a time corresponding to a time prior to t ₄, such as t ₂ or t ₃.

In some embodiments, where the appearance of the user interface element is based on a gradient between colors sampled from portions of the environment behind the user interface element, the colors are sampled from one or more other computer-generated user interface elements, and the appearance of the elements is also based on a corresponding gradient between colors sampled from portions of the environment behind the other computer-generated elements, and updated over time, similar to the foreground user interface element. For example, if in fig. 7H-7J the appearance of one or more regions of wall-hanging 7106 is based on a color gradient between multiple colors sampled from one or more portions of environment 7104 behind wall-hanging 7106 (e.g., one or more portions of wall 7004'), then the appearance of user interface element 7040 would be based on the color sampled from another computer-generated object (wall-hanging 7106), the appearance of which is also updated over time according to the techniques described above with reference to user interface element 7040 in fig. 7H-7J.

In some cases, multiple user interface elements whose appearance is based on overlapping portions of the three-dimensional environment (e.g., based on graphical composition and/or color gradient) are displayed to at least partially overlap one another (e.g., overlapping user interface elements 7050 and 7052 as described herein with reference to fig. 7K). In some implementations, the appearance of the user interface element in the background is generated (e.g., based on graphical composition and/or color gradient) using one or more of the techniques described herein, and the appearance of the portion of the user interface element in the foreground that overlaps the user interface element in the background is based on, defined by, or the same as the appearance of the corresponding portion of the underlying user interface element. For example, as described above, if the appearance of wall hanging 7106 is based on a color gradient between the color sampled from wall hanging 7004' and the other underlying portions of environment 7104 subsequent to wall hanging 7106, the appearance of the portion of user interface element 7040 overlaid on wall hanging 7106 is optionally based on, the same as, or a copy of the corresponding underlying portion of wall hanging 7106, thereby reducing the computational burden in generating user interface element 7040.

In some implementations, the appearance of the user interface element is based on a color gradient between colors sampled from portions of the environment that overlap (although not necessarily behind) the user interface element. For example, as described herein with reference to fig. 7F, portions of the environment 7104 that are closer to the viewpoint 7032 of the user than the user interface element 7030 to the viewpoint 7032 of the user may contribute to the appearance of the user interface element 7030. Similarly, the simulated position of user interface element 7040 in environment 7104 may be farther from the viewpoint of the user in fig. 7H-7J than box 7012', wall-hanging 7106, and/or 7014', while the color gradient of the appearance of user interface element 7040 may be generated from colors sampled from box 7012', wall-hanging 7106, and/or 7014', and optionally one or more other portions of environment 7104 that are closer to the viewpoint of the user than the simulated position of user interface element 7040 (e.g., if the simulated position of user interface element 7040 in fig. 7H-7J is position 7030-4 relative to environment 7104, as shown in fig. 7F).

Fig. 7K-7O illustrate user interface elements (e.g., user interface element 7050 and user interface element 7052) displaying an appearance of a simulated material having a simulated thickness and curvature along edges of the user interface element, wherein the edges of the user interface element have simulated optical properties that interact with light in a three-dimensional environment, according to some embodiments. The user interfaces in fig. 7K-7O are used to illustrate the processes described below, including the processes in fig. 10 and 11. According to some embodiments, the user interface elements described with reference to fig. 7K-7O optionally have one or more of the characteristics and/or behaviors of the user interface elements described herein with respect to other examples (such as examples in fig. 7A-7G, 7H-7J, 7P, 7Q-7W, and/or 7X-7Z), unless otherwise indicated in the specification.

Fig. 7K shows a view of the environment 7104 displayed via the display generating section 7100. The displayed environmental view 7104 includes a view of a portion of box 7012', a view of wall-hanging 7106, and a view of a portion of box 7014'. Fig. 7K illustrates a first user interface element 7050 and a second user interface element 7052 displayed over a view of environment 7104. The user interface element 7050 in fig. 7K is displayed as a three-dimensional object (e.g., a pane or tray of simulated material, such as a transparent or translucent material) having an appearance made of simulated material. Further, the user interface element 7052 in fig. 7K is displayed as a three-dimensional object (e.g., a pane or tray of simulated material, such as a transparent or translucent material) having an appearance made of simulated material. The user interface element 7050 has an appearance based on an overlapping portion of the environment 7104 (such as the wall 7004 'and/or the box 7012'), a graphical composition using frames from different times as described herein with reference to fig. 7A-7G, a color gradient as described with reference to fig. 7H-7J, or any other exemplary technique described herein. Additionally, in some implementations, the appearance of the user interface element 7050 is updated over time based on changes in the appearance of overlapping portions of the environment 7104, similar to the user interface elements described herein with reference to fig. 7A-7J. Similarly, user interface element 7052 has an appearance based on underlying portions of environment 7104 (such as walls 7004', boxes 7012', and/or wall hanging 7106), a graphical composition using frames from different times, a color gradient, or other exemplary techniques described herein. In some embodiments, the appearance of the user interface element 7050 is also updated over time based on changes in the appearance of the overlapping portions of the environment 7104, similar to the user interface elements described herein with reference to fig. 7A-7J and the user interface element 7050.

In the example of fig. 7K, from the user's point of view, the user interface element 7050 is in front of the user interface element 7052 in the environment 7104 (e.g., the user interface element 7050 is positioned closer to the user's point of view than the user interface element 7052, and the user interface element 7050 optionally obscures a portion of the left side of the user interface element 7052). Further, from the user's point of view, user interface element 7052 is in front of, and optionally obscures, boxes 7012', wall-hanging 7106, and wall 7004 '(e.g., user interface element 7052 is positioned closer to the user's point of view than boxes 7012', wall-hanging 7106, and wall 7004'). As described herein with reference to fig. 7H-7J, in fig. 7K, the appearance of the portion of the foreground user interface element 7050 that overlaps the background user interface element 7052 is optionally separate from the appearance of the corresponding portion of the user interface element 7052 that is generated based in part on the wall 7004 'and the box 7012', based on the corresponding underlying portion of the user interface element 7052, the same as, or a copy of, it (e.g., the appearance of the overlapping portion of the user interface element 7050 is not generated independently based on the appearance of the wall 7004 'and the box 7012').

In addition, FIG. 7K shows a representation of an edge 7050-e of the user interface element 7050. The edges 7050-e represent that the user interface element 7050 is displayed with a simulated thickness between the front surface of the user interface element 7050 and the back surface of the user interface element 7050. The edges 7050-e are shown in fig. 7K as being curved in two dimensions (e.g., the shape of the user interface element 7050 is rounded rectangle from the user's perspective). Alternatively or in addition, the edge 7050-e represents a three-dimensional curvature (e.g., no straight edges resulting from the angled intersection of two faces). Alternatively, the edges 7050-e represent multiple planes (e.g., the planes of a rectangular prism), such as the top, bottom, and side surfaces of the user interface element 7054 in fig. 7L. In some embodiments, the edges 7050-e of the user interface element 7050 are displayed with a visual effect that reflects light from one or more light sources in the environment 7104. For example, a reflection (sometimes referred to as a specular reflection) is optionally displayed at a first location on the edge 7050-e to simulate a reflection from an outer surface of the user interface element 7050. In some implementations, reflections are optionally displayed at a second location on the edge 7050-e opposite the first location (e.g., a lower right corner opposite an upper left corner, and vice versa) to simulate internal reflections from the inner surface of the user interface element 7050 (e.g., internal reflections of some light that were not reflected at the location of the external specular reflection and have traveled through to the opposite side of the user interface element 7050). In some implementations, the amount of specular reflection displayed along the edge 7050-e is based on the simulated thickness of the edge 7050-e of the user interface element 7050 (e.g., thinner panes reflect less light than thicker panes).

Similarly, the edge 7052-e represents that the user interface element 7052 is displayed with a simulated thickness between the front surface of the user interface element 7052 and the back surface of the user interface element 7052. While shown as a two-dimensional bend in fig. 7K, the edges 7052-e of the user interface element 7052 optionally can be three-dimensional bends or alternatively represent multiple planes, such as the top, bottom, and side surfaces of the user interface element 7054 in fig. 7L. Furthermore, in some embodiments, edges 7052-e of user interface element 7052 are displayed with a visual effect that reflects light from one or more light sources in environment 7104. For example, a specular reflection is optionally displayed at a first location on the edge 7052-e to simulate reflection from an outer surface of the user interface element 7052, and another specular reflection is optionally displayed at a second location on the edge 7052-e opposite the first location (e.g., a bottom edge opposite the top edge, and vice versa) to simulate internal reflection from an inner surface of the user interface element 7052 (e.g., internal reflection of some light that is not reflected at the location of the external specular reflection and has traveled through to the opposite side of the user interface element 7052). In some implementations, the amount of specular reflection displayed along the edge 7052-e is based on the simulated thickness of the edge 7052-e of the user interface element 7052 (e.g., thinner panes reflect less light than thicker panes).

In some implementations, specular reflection of the user interface elements 7050 and/or 7052 in fig. 7K simulates reflection of light from one or more light sources in the environment 7104. In some embodiments, the one or more light sources include one or more physical light sources detected in a physical environment (e.g., environment 7102 in fig. 7A) corresponding to environment 7104. In some embodiments, the one or more light sources include one or more analog light sources in environment 7104. Simulated specular reflection from a computer-generated user interface object is described in more detail herein with reference to fig. 7L-7O.

In some implementations, the appearance of the user interface element (including any specular reflection) changes based on different contexts of the three-dimensional environment. For example, the appearance of the user interface element 7050 and/or the appearance of the user interface element 7052 changes based on different contexts of the environment 7104, such as different times of day (e.g., dawn or dusk versus noon, sunlight versus artificial light in the physical environment 7102 (fig. 7A) corresponding to the environment 7104), different room color temperatures (e.g., warmer hues versus cooler hues in the environment 7104), different brightness (e.g., brightness of light and/or color in the environment 7104), and other environmental characteristics of the environment 7104.

As described above, the user interface element 7050 is displayed, for example, as a pane of simulated transparent or partially transparent material. In some implementations, content associated with the user interface element 7050 is displayed. In some implementations, the content is displayed with a visual effect positioned behind the front surface of the user interface element 7050 embedded within a pane of simulated material (e.g., between the front surface and the back surface of the user interface element 7050 at a depth less than the simulated thickness of the user interface element 7050). In some implementations, the content of the user interface element 7050 proximate to (e.g., within a threshold distance of) the edge 7050-e is displayed with a visual effect of bending or wrapping around the simulated thickness of the edge 7050-e to simulate the appearance of internal reflection and/or refraction within the user interface element 7050 of the content (e.g., as would occur at the edge of a lens or glass pane). The content is configured to be moved (e.g., scrolled, repositioned, resized, rescaled, rotated, and/or other transformations applied) in response to the user input. As the content moves, different portions of the content are each within a threshold distance of the edge 7050-e. Thus, the optical effect of internal reflection and/or refraction is updated based on the updated portions of content that are within the threshold distance of the edge 7050-e.

Similarly, in some embodiments, the user interface element 7052 is displayed as a pane of simulated transparent or partially transparent material. In some implementations, the content associated with the user interface element 7052 is displayed with a visual effect positioned behind the front surface of the user interface element 7052 embedded within a pane of simulated material. In some implementations, the content of the user interface element 7052 proximate to the edge 7052-e of the user interface element 7052 is displayed with a visual effect of bending or wrapping around the simulated thickness of the edge 7052-e and updated as the content moves such that different portions of the content all move within a threshold distance of the edge 7052-e.

In some embodiments, content displayed as embedded within the user interface element is visible from the front of the user interface element and is not apparent or visible at all from the back of the user interface element. For example, fig. 7K shows a front view of user interface elements 7050 and 7052. In some embodiments, from the perspective of the user interface elements 7050 and 7052, the content displayed as embedded within the user interface element 7050 and/or the user interface element 7052 is visible, as shown in fig. 7K. If the user's point of view changes relative to the user interface element 7050 to a view of the back of the user interface element 7050 (e.g., movement of the user interface element 7050 to the back of the user interface element 7050 via the user's point of view moves or flips or rotates the user interface element 7050), then in some embodiments the content is visually impaired, such as by blurring, fading, shading, or darkening, or the content is not displayed. The user interface element 7052 optionally exhibits similar behavior to the user interface element 7050.

In fig. 7K, user interface elements 7050 and 7052 cast one or more shadows on other objects in environment 7104. As described in more detail herein with respect to fig. 7L-7O, in some embodiments, shadows cast by user interface elements in a displayed three-dimensional environment are based on one or more light sources in the three-dimensional environment, such as simulated or computer-generated light sources and/or physical light sources in a physical environment corresponding to the displayed three-dimensional environment. In FIG. 7K, the user interface element 7050 casts shadows 7050-s on the user interface element 7052 behind the user interface element 7050. The user interface element 7052 casts shadows 7052-s on the box 7012', wall-mount 7106, and wall 7004' behind the user interface element 7052.

Fig. 7L illustrates a user interface element 7054 displayed over a view of environment 7104. The user interface element 7054 is shown having a thickness that is visible along the top and left edges of the user interface element 7054 from the user's point of view in fig. 7L. Furthermore, where the user interface element 7054 is displayed as a transparent or partially transparent material, the thickness of the user interface element 7054 is optionally visibly apparent from the right and bottom edges, which would be displayed as partially visible through the user interface element 7054 from the viewpoint of the user in fig. 7L.

In addition, the user interface element 7054 includes a plurality of regions. Specifically, the user interface element 7054 includes a first region 7054-a (e.g., a pane or tray of simulated material) that defines a front surface (e.g., a front face or front plane) of the user interface element 7054. In some embodiments, the region 7054-a of the user interface element 7054 is similar to the background portion 7030-a of the user interface element 7030 (fig. 7D) and/or the background portion 7040-a of the user interface element 7040 (fig. 7H). The user interface element 7054 also includes regions 7054-b, 7054-c, and 7054-d that have different depths relative to a front surface defined by the user interface element 7054-a. In some embodiments, the region 7054-a is a background portion of the user interface element 7054. In some embodiments, the regions 7054-a to 7054-c are content regions that include visual content such as text or graphics, or selectable and/or activatable objects such as buttons or controls. As shown in fig. 7L, the regions 7054-b and 7054-c have a concave appearance in the front surface of the user interface element 7054 (e.g., have the appearance of a recess in the surface, or have the appearance of a portion of material removed from the surface, such as by etching or cutting). In contrast, the regions 7054-d have the appearance of material that is raised relative to the front surface of the user interface element 7054 (e.g., the appearance of portions of material that are extruded, embossed, added to or placed on top of the surface, or lifted off the surface). In some embodiments, the user interface element includes one or more raised regions similar to raised regions 7054-d. In some implementations, the raised areas of the user interface element 7054 correspond to user interface elements, such as buttons, menus, or other controls, that can be activated to perform operations in the user interface or in the environment 7104. In some implementations, the recessed area of the user interface element 7054 corresponds to a user interface element that is a content input area (such as a text input box).

The user interface element 7054 is displayed as a material having optical properties that interact with light in the environment 7104. Furthermore, different regions of the user interface element 7054 interact with light in the environment 7104 according to the characteristics of those regions and according to the source of the light. Fig. 7L shows two light sources, a light source 7056 and a light source 7058. The light source 7056 may be a physical light source (e.g., physically present in the physical environment 7102 (fig. 7A)) or an analog light source in the environment 7104. Similarly, the light source 7058 may be a physical light source or an analog light source. For example, both light sources 7056 and 7058 may be physical light sources. Both light sources 7056 and 7058 may be analog light sources; or one light source may be a physical light source and the other a simulated light source.

As shown in fig. 7L, the user interface element 7054 is displayed with an appearance that reflects light from the light source 7058 as it appears to a person viewing the user interface element 7054 at a location in the environment 7104 in fig. 7L and from the viewpoint of the user in fig. 7L. For example, the top edge of the user interface element 7054 faces the light source 7058 and is shown as having a reflection (sometimes referred to herein as specular reflection) that corresponds to the light source 7058. In some implementations, the light source 7058 is a physical light source in the physical environment 7102 and specular reflection from a top edge of the user interface element 7054 is generated and displayed by a computer system to simulate the appearance of a physical object in the physical environment 7102 reflecting light from the light source 7058. In some implementations, the light source 7058 is a simulated light source in the environment 7104 and also simulates specular reflection from the top edge of the user interface element 7058, as described herein. In some implementations, the amount of specular reflection from the top edge of the user interface element 7054 is based on the simulated thickness of the user interface element 7054.

In another example, the raised regions 7054-d reflect light along their top edges that face the light sources 7058 in a similar manner to the top edges of the user interface 7054. In the example shown in fig. 7L, the amount of specular reflection from the top edge of the raised area 7054-d is less than the amount of specular reflection from the top edge of the user interface element 7054 because the thickness of the raised area 7054-d relative to the front surface of the user interface element 7054 (e.g., the extent to which the raised area 7054-d protrudes from the surface of the user interface element 7054) is less than the thickness of the top edge of the user interface element 7054.

In contrast to the top edges of the user interface element 7054 and the raised regions 7054-d, the recessed regions 7054-b and 7054-c reflect light along their bottom edges because the bottom edges of the recessed regions 7054-b and 7054-c face the light source 7058. Although the depth of the recessed regions 7054-b and 7054-c relative to the front surface of the user interface element 7054 is the same as the thickness of the raised regions 7054-d relative to the front surface of the user interface element 7054, the amount of specular reflection from the bottom edges of the recessed regions 7054-b is less than the amount of specular reflection from the top edges of the raised regions 7054-c. This is consistent with the following principle: because the light must travel through the simulated material of the user interface element 7054, less light reaches the bottom edge of the recessed region 7054-b, in contrast to relatively less blockage by light reaches the top edge of the raised region 7054-c. The amount of specular reflection from the bottom edge of the recessed region 7054-c is again less than the amount of specular reflection from the bottom edge of the recessed region 7054-b, consistent with the following principles: for example, because light must travel a longer distance through the simulated material of the user interface element 7054 (e.g., and past the raised region 7054-c), light reaching the bottom edge of the recessed region 7054-b will be more blocked than light reaching the bottom edge of the recessed region 7054-b.

In some embodiments, the user interface element 7054 is also displayed with the appearance of light reflected into the user interface element 7054 from an edge of the user interface element 7054 opposite the light source. For example, fig. 7L illustrates specular reflection from a bottom edge of the user interface element 7054 as a function of internal reflection of light traveling from the light source 7058 through the user interface element 7054 and as a function of the user interface element 7054 being displayed as a transparent or partially transparent material (e.g., internal specular reflection is shown along an inner side of an edge of the user interface element 7054 opposite a location of external specular reflection). As shown in fig. 7L, the amount of specular reflection from the bottom edge of the user interface element 7054 is less than the amount of specular reflection from the top edge of the user interface element 7054, depending on whether the specular reflection from the bottom edge of the user interface element 7054 is associated with internal reflection and/or depending on whether the bottom edge of the user interface element 7054 is farther from the light source 7058 than the top edge of the user interface element 7054.

While the user interface element 7054 and its constituent regions are generally shown as having rectangular corners (also referred to as vertices) and rectangular edges (e.g., the regions 7054-a through 7054-d are shown as rectangular prisms having rectangular depressions), one of ordinary skill in the art will recognize that other three-dimensional shapes are possible. For example, the corners of the user interface element 7054 and/or the regions 7054-a-7054-d are optionally rounded rather than rectangular as shown in fig. 7L-7O (e.g., as viewed from the user having a rounded rectangular shape, e.g., the top surface of the user interface element 7054 curves downward and becomes the side surface of the user interface element 7054, rather than intersecting the right surface at one edge and the left surface at the other edge, similar to the shape of the user interface elements 7050 and 7052 in fig. 7K). Alternatively or in addition, the edges of the user interface element 7054 and/or the regions 7054-a to 7054-d are optionally rounded rather than rectangular as shown in fig. 7L to 7O (e.g., having a chamfer or curvature along the perimeter of the user interface element 7054, e.g., the front face of the user interface element 7054 corresponding to region 7054-a curves back away from the user and becomes the side face of the user interface element 7054, rather than intersecting the right face at one edge and the left face at the other edge). In some implementations, specular reflection on the user interface element 7054 is positioned according to the shape of the surface of the user interface element at corners and edges (e.g., specular reflection is shown for rounded corners as opposed to for rectangular corners).

In addition to specular reflection, the user interface element may cast shadows on other objects. For example, in FIG. 7L, the raised region 7054-d casts shadows 7054-s2 onto the user interface element 7054 according to the raised region 7054-d blocking some of the light from the light source 7058 from reaching portions of the user interface element 7054 that are below the raised region 7054-d and slightly to the left. According to the user interface element 7054 blocking some of the light from the light source 7058 from reaching a portion of the environment 7104 that is below the user interface element 7054 and slightly to the left, the user interface element 7054 is also shown to cast shadows 7054-s1 in the environment 7104. Those of ordinary skill in the art will recognize that similar principles regarding specular reflection and/or shadows may be applied to other objects in environment 7104, including but not limited to other computer-generated objects and/or representations of physical objects (e.g., simulated light may be projected onto a physical object, thereby producing a simulated specular reflection, and a physical object may cast a simulated shadow).

In some implementations, specular reflection displayed from objects in the environment 7104 is based on a subset of light sources in the environment 7104 and/or the corresponding physical environment 7102. One of ordinary skill in the art will recognize that one or more light sources may be selectively ignored in generating and rendering a view of environment 7104. For example, although two light sources 7056 and 7058 are shown in fig. 7L, the specular reflection shown in fig. 7L is based on the light source 7058 and not on the light source 7056. In some cases, light source 7506 is a physical light source, light source 7508 is an analog light source, and specular reflection is based on light source 7508, with physical light source 7506 being ignored. This may provide the user with a more immersive viewing experience of environment 7104 in which the light sources are entirely generated by the computer system. Alternatively, the light source 7508 is a physical light source and the display of specular reflection based on the physical light source 7508 improves user security by providing the user with a viewing experience that is more closely related to the user's surrounding physical environment.

Fig. 7M illustrates a user interface element 7054 displayed at a different location relative to the environment 7104. In fig. 7M, the user interface element 7054 is closer to the viewpoint of the user than in fig. 7L. Accordingly, the user interface element 7054 is shown in greater scale (e.g., wider and higher) in fig. 7M than in fig. 7L and has a thickness h ₂ that is greater than the thickness h ₁ of the user interface element 7054 in fig. 7L. According to what appears to be a larger object when positioned closer to the user, the increase in thickness h ₂ of the user interface element 7054 in fig. 7M is due in part to the repositioning of the user interface element 7054 closer to the viewpoint of the user. Further, in some embodiments, the thickness of the user interface element 7054 is increased by an amount greater than an amount associated with repositioning the user interface element 7054. For example, if repositioning the object from the position shown in fig. 7L to position 7M is associated with a first amount of increase in the display thickness of the object (e.g., user interface element 7054 would be displayed having a thickness h ₃ that is greater than h ₁ and less than h ₂), then user interface element 7054 is displayed having a second amount of increase in its display thickness, wherein the second amount is greater than the first amount. In some embodiments, enlarging the increase in display thickness of objects that have been moved closer to the user in the displayed three-dimensional environment helps the user better identify the repositioned objects as being closer.

Fig. 7M also includes a side view 7055 that illustrates an outline of the user interface element 7054. The side view 7055 illustrates the thicknesses of the recessed regions 7054-b, recessed regions 7054-c, and raised regions 7054-d relative to the user interface element 7054 (e.g., the depths of the recessed regions 7054-b and 7054-c relative to the front surface of the user interface element 7054, and the thicknesses of the raised regions 7054-d relative to the front surface of the user interface element 7054).

Further, in fig. 7M, the specular reflection on the user interface element 7054 is located at a different position on the user interface element 7054 than in fig. 7L. Specifically, the specular reflection in fig. 7M is based on the light source 7056 instead of the light source 7058 as in fig. 7L, and simulates a reflection from the light source 7056 off of the user interface element 7054 as it would appear to a person viewing the user interface element 7054 at a location in the environment 7104 in fig. 7M and from the viewpoint of the user in fig. 7M. For example, fig. 7M shows specular reflection at the upper left corner (e.g., top edge) of the user interface element 7054 (e.g., at a location along the outer edge of the user interface element 7054 that faces the light source 7056), and specular reflection at the upper left corner of the raised region 7054-d (e.g., at a location along the outer edge of the raised region 7054-d that faces the light source 7056). Specular reflection is also shown at the lower right-inner corner of the recessed region 7054-b (e.g., at a location along the edge of the recessed region 7054-b opposite the light source 7056) and at the lower right-inner corner of the recessed region 7054-c (e.g., at a location along the edge of the recessed region 7054-c opposite the light source 7056). The user interface element 7054 is also displayed with the appearance of light reflected into the user interface element 7054 from its lower right corner (e.g., internal specular reflection is shown along the inside of the edge of the user interface element 7054 opposite the location of external specular reflection), with the amount of internal specular reflection at the lower right corner being less than the amount of external specular reflection at the upper left corner.

In some cases, the change in the position of the specular reflection from the user interface element 7054 from the position shown in fig. 7L to the position shown in fig. 7M is due at least in part to the change in the position of the user interface element 7054 (e.g., in accordance with the user interface element 7054 moving farther from and/or less directly below the spatial position in the environment 7104 of the light source 7058, and the spatial position in the environment 7104 that is closer to the light source 7056 moving and/or more directly below it). In some cases, the change in the position of the specular reflection from the user interface element 7054 is due, at least in part, to a change in the user's point of view relative to the user interface element 7054. In some cases, the change in the position of the specular reflection from the user interface element 7054 from the position shown in fig. 7L to the position shown in fig. 7M is due, at least in part, to a change in the light source 7056 and/or the light source 7058 (e.g., the light source 7058 has been reduced in brightness or turned off in fig. 7M and/or the light source 7056 has been increased in brightness or turned on in fig. 7M as compared to fig. 7L). In some embodiments, the computer system selects which light source the specular reflection is based on, for example, due to contextual factors such as time of day (e.g., daylight may be used as the light source during the day and a light may be used as the light source during the night), room color temperature (e.g., a warmer color temperature light source may be used for specular reflection in a room having a warmer hue and/or to achieve a warmer atmosphere in environment 7104), brightness (e.g., a brighter light source may be used for specular reflection in a brighter room, or to achieve a brighter room), and/or in response to a user selection of one or more particular light sources.

As discussed herein with reference to fig. 7L, the user interface element 7054 and portions thereof can cast shadows on other objects. For example, in FIG. 7M, the raised regions 7054-d cast shadows 7054-s3 onto the user interface element 7054 according to the raised regions 7054-d blocking some of the light from the light sources 7056 from reaching portions of the user interface element 7054 that are below and to the right of the raised regions 7054-d. The user interface element 7054 is also shown as casting shadows 7054-s4 in the environment 7104 according to the user interface element 7054 blocking some of the light from the light source 7056 from reaching portions of the environment 7104 that are below and to the right of the user interface element 7054.

As also discussed herein with reference to fig. 7L, in some embodiments, specular reflection from the user interface element 7054 is based on a subset of the light sources in the environment 7104 and/or the corresponding physical environment 7102. For example, the specular reflection shown in fig. 7M is based on the light source 7056 rather than the light source 7058 (as is the case in fig. 7L). One of ordinary skill in the art will recognize that other embodiments are possible in which specular reflections of both the light source 7056 and the light source 7058 are displayed on the user interface element 7054 and/or other objects in the environment 7104.

Fig. 7N illustrates a transition made in the user interface element 7054 in response to a user interaction with the user interface element 7054. Fig. 7N illustrates that the user 7002 is focusing (e.g., gazing) on the raised region 7054-d, indicated in fig. 7N by dashed lines extending from the eyes of the user 7002 to the raised region 7054-d. In response to the user 7002 focusing on the raised region 7054-d, the raised region 7054-d protrudes more from the surface of the user interface element 7054 (e.g., from the front surface defined by region 7054-a). Side view 7055 shows a profile illustrating an increase in thickness (e.g., as compared to side view 7055 in fig. 7M) of raised regions 7054-d relative to the surface of user interface element 7054 without any change in thickness (e.g., depth) of recessed regions 7054-b and 7054-c. According to the increased thickness of the raised region 7054-d, a greater amount of specular reflection from the upper left corner of the raised region 7054-d is illustrated and the raised region 7054-d is shown to cast a longer shadow onto portions of the user interface element 7054 that are below and to the right of the raised region 7054-d.

In some embodiments, in response to the user focusing attention on the raised area, the thickness of the raised area of the user interface element relative to the surface of the user interface element is changed (e.g., the thickness of raised area 7054-d increases, as shown in fig. 7N; alternatively, the thickness is reduced). In some implementations, the thickness of the raised region of the user interface element changes a first amount in response to a first portion of user input focusing attention on the raised region (e.g., increases the thickness in response to gazing at the raised region), and then changes a second amount in response to a second subsequent portion of user input (e.g., decreases the thickness in response to selection or activation of the raised region or more specifically a control defined by the raised region). In some implementations, the depth of the recessed region of the user interface element does not change in response to the user focusing on the recessed region (e.g., the depth of the recessed region 7054-b or 7054-c will not change in response to the user's gaze). Alternatively, in some embodiments, in response to a user focusing attention on a recessed region, the depth of the recessed region of the user interface element is changed (e.g., the depth is increased, or the recessed region is changed to a raised region), as described in more detail herein with reference to fig. 7O.

In some embodiments, the raised area of the user interface element comprises a plurality of layers, each layer raised to a different degree relative to the surface of the user interface element. For example, while the raised regions 7054-d are shown in fig. 7L-7M as having a first thickness relative to the front surface of the user interface element 7054 and a second, greater thickness relative to the front surface of the user interface element 7054 in fig. 7N, one of ordinary skill in the art will recognize that the raised regions 7054-d may comprise a stack of layers each having a corresponding thickness. User interface elements having different layers and interactions with such user interface elements are described in more detail herein with reference to fig. 7P.

Fig. 7O illustrates a transition made in the user interface element 7054 in response to a user interaction with the user interface element 7054 that is different from the interaction illustrated in fig. 7N. Fig. 7O illustrates that the user 7002 is focusing (e.g., gazing) on the recessed region 7054-b, indicated in fig. 7O by a dashed line extending from the eyes of the user 7002 to the recessed region 7054-b (e.g., transitioning from fig. 7M due to the user 7002 focusing on the recessed region 7054-b, or transitioning from fig. 7N due to the user 7002 diverting attention from the region 7054-d to the region 7054-b). In response to the user 7002 focusing on the recessed region 7054-b, the recessed region 7054-b is recessed more from the surface of the user interface element 7054 (e.g., from the front surface defined by region 7054-a). Side view 7055 shows a profile illustrating an increase in depth of recessed regions 7054-b relative to the surface of user interface element 7054 (e.g., as compared to side view 7055 in fig. 7M) without any change in thickness of recessed regions 7054-c and without any change in thickness (e.g., height) of raised regions 7054-d. Depending on the increased depth of the recessed region 7054-b, a greater amount of specular reflection from the inner lower right corner of the recessed region 7054-b is shown (e.g., specular reflection from the recessed region 7054-c and the raised region 7054-d does not change any). In the example of fig. 7O, the change in specular reflection from the recessed region 7054-b is due to a change in the user interface element 7054 (e.g., more generally, a change in the simulated optical properties of the simulated material forming the recessed region 7054-b and/or the user interface element 7054) rather than due to a change in the simulated and/or physical light source(s) in a three-dimensional environment (as described herein, this may also be considered in some embodiments). In some embodiments, the user interface element 7054 is shown as casting a shadow onto a recessed region such as 7054-b and/or 7054-c (e.g., a shadow cast by an edge of the user interface element 7054 that adjoins the recessed region), and optionally, a greater amount of shadow is shown as being cast by the user interface element 7054 onto the recessed region 7054-b as the depth of the recessed region 7054-b increases.

In some embodiments, in response to the user focusing attention on the recessed region, the depth of the recessed region of the user interface element relative to the surface of the user interface element is changed (e.g., the depth of recessed region 7054-b increases, as shown in fig. 7O; alternatively, the depth decreases). In some implementations, the depth of the recessed region of the user interface element changes a first amount in response to a first portion of user input focusing on the recessed region (e.g., increases the depth in response to gazing at the recessed region), and then changes a second amount in response to a second subsequent portion of user input (e.g., decreases the depth in response to selection or activation of the recessed region or more specifically a control defined by the recessed region). In some implementations, the depth of the recessed region of the user interface element does not change in response to the user focusing on the recessed region (e.g., the depth of the recessed region 7054-b or 7054-c will not change in response to the user's gaze). Alternatively, in some embodiments, the depth of the recessed region of the user interface element is changed by changing the recessed region to a raised region in response to the user focusing on the recessed region.

In some cases, such as those embodiments in which the recessed area is an input field (such as for text input), the recessed area includes text. For example, a recessed area for a search bar may include a text label, such as the word "search," as a visual cue as to the purpose of the recessed area and how to interact with the recessed area. In another example, the recessed area for the input field includes text previously entered by the user (e.g., entered search terms or search queries provided via a voice command, keyboard, or other input device). Using the examples of fig. 7M-7N, in some embodiments, when the user is not focused on the recessed region 7054-b, the text is positioned at a first depth relative to the recessed region 7054-b. In some implementations, the text is at the same plane as the surface into which the region is recessed (e.g., zero depth relative to the user interface element 7054, at the same simulated distance from the user's point of view as the front surface of the user interface element 7054 defined by region 7054-a). In some embodiments, the text is within the recessed region beyond the plane of the surface into which the region is recessed (e.g., at a greater simulated distance from the user's viewpoint than the front surface of the user interface element 7054 relative to the non-zero depth of the user interface element 7054). In some embodiments, the text is in front of (e.g., at a lesser depth than) the recessed surface of region 7054-b. In some embodiments, as the viewpoint of the user 7002 moves, the text exhibits parallax relative to the recessed surface, which increases the perception of depth by the user within the environment 7104.

In some embodiments, in response to the user 7002 focusing attention on the recessed region 7054-b, as described herein with reference to fig. 7O, in conjunction with a change (e.g., an increase) in the depth of the recessed region 7054-b, the depth of text in the recessed region 7054-b is also changed (e.g., increased), optionally by an amount corresponding to the amount of change in the depth of the recessed region 7054-b. In some implementations, in response to the user 7002 focusing on the recessed region 7054-b (e.g., merely indicating that an interaction is ready), the depth of text in the recessed region 7054-b does not change until the user 7002 interacts with the recessed region 7054-b. In some such implementations, in response to a user 7002 interacting with the recessed region 7054-b, such as with an input gesture similar to the input gesture of fig. 7U described herein, the depth of text in the recessed region 7054-b is changed (e.g., increased) in response to a first portion of the input gesture, optionally by an amount corresponding to an amount of change in the depth of the recessed region 7054-b that occurred previously in response to the user 7002 focusing attention on the recessed region 7054-b (e.g., the change in depth of text is delayed relative to the change in depth of the recessed region 7054-b). In some implementations, the depth of the recessed region 7054-b does not change further in response to the first portion of the input gesture. in some implementations, in response to the first portion of the input gesture, the depth of text in the recessed region 7054-b is further changed (e.g., further increased) beyond what was previously occurring (if any) in response to the user 7002 focusing on the recessed region 7054-b, optionally while maintaining the depth of the recessed region 7054-b. In some implementations, the change in depth of text in the recessed region 7054-b is reversed in response to the end of the input gesture. For example, in response to the end of the input gesture, the text reverts to the same depth as in response to the first portion of the input gesture, the same depth as in response to the user 7002 focusing on the recessed region 7054-b, a default depth just before the user 7002 focuses on the recessed region 7054-b, or any other depth that is different (e.g., less) than the depth of the text in the recessed region 7054-b in response to the first portion of the input gesture. In some implementations, as long as the user 7002 remains focused on the recessed region 7054-b, the change in text depth is partially reversed to an intermediate depth (e.g., the same depth as in response to the user 7002 focusing on the recessed region 7054-b) after the input gesture ends, and then is further reversed in response to the user 7002 no longer focusing on the recessed region 7054-b. In some implementations, the changed depth of the recessed region 7054-b is maintained after the input gesture ends as long as the user 7002's attention remains focused on the recessed region 7054-b. in some implementations, the change in text depth is completely reversed in response to the end of the input gesture (e.g., regardless of whether the user 7002 is attentive and even though it is still focused on the recessed region 7054-b). In some implementations, the change in depth of the recessed region 7054-b is also reversed in response to the end of the input gesture (e.g., regardless of whether the user 7002 is attentive and even though it is still focused on the recessed region 7054-b).

Fig. 7P illustrates separating layers of a user interface element (e.g., user interface element 7060) to different degrees in response to different user interactions that include a user focusing attention on the user interface element, according to some embodiments. The user interface in fig. 7P is used to illustrate the processes described below, including the process in fig. 12. According to some embodiments, the user interface elements described with reference to fig. 7P optionally have one or more of the characteristics and/or behaviors of the user interface elements described herein with respect to other examples (such as examples in fig. 7A-7G, 7H-7J, 7K-7O, 7Q-7W, and/or 7X-7Z), unless otherwise indicated in the specification.

In fig. 7P, scene 7062 shows a view of computer system 101 displaying environment 7104 (e.g., via display generation component 7100) that includes a representation of a portion of physical environment 7102 (fig. 7A), as described herein. Further, displayed in environment 7104 in fig. 7P are a plurality of user interface elements (also referred to herein as icons). The plurality of user interface elements are displayed as an icon group 7070 and represent a master user interface or launch pad for the computer system 101, such as an icon for launching an application, an icon for starting a communication session (e.g., an icon corresponding to a different user than the user 7002), an icon for starting a computer-generated experience, a group icon for a different class of user interface objects, a container object such as a folder or group, a master button, a dock, a root menu, or other user interface element for performing operations in and interacting with a three-dimensional environment. The respective icons in the icon group 7070 can be activated to perform one or more operations (e.g., launch an application or experience, open a file or folder, etc.) in the environment 7104.

Fig. 7P also shows an enlarged view 7072 of the respective icon 7060, which illustrates that the icon 7060 includes a plurality of icon elements associated with different icon layers. For example, the envelope graphic is associated with (e.g., assigned to) the foreground layer 7060-1; the circular element is associated with (e.g., assigned to) middle layer 706-2; and square backing elements are associated with (e.g., assigned to) the background layer 7030-3. Those of ordinary skill in the art will recognize that additional elements (e.g., more than one element at a time) may be associated with respective layers of the icon 7060, and that the user interface element 7060 may include more or fewer layers.

Fig. 7P also shows icons 7060 with different degrees of separation between layers 7060-1, 7060-2, and 7060-3. For example, view 7074 shows icon 7060 without separation between layers 7060-1 to 7060-3. View 7076 shows an icon 7060 with a first amount (or degree) of separation between each of the layers 7060-1 to 7060-3. View 7078 shows an icon 7060 having a second separation (or degree) that is greater than the first separation between each of the layers 7060-1 to 7060-3. One of ordinary skill in the art will recognize that different amounts of separation may be used for different groups of layers (e.g., optionally, the amount of separation between layers 7060-1 and 7060-2 is different than the amount of separation between layers 7060-2 and 7060-3 in views 7076 and/or 7078).

In some embodiments, the layers of the user interface element cast light and/or shadows onto one or more other layers of the user interface element. For example, as described herein with reference to fig. 7N, a raised area of a user interface element (e.g., corresponding to an interactive control or content input area), such as raised area 7054-d of user interface element 7054, optionally casts shadows onto user interface element 7054 and/or other portions of environment 7104. Furthermore, as described herein with reference to fig. 7N, the raised regions of the user interface element optionally include stacks of layers each having a respective thickness. For example, as shown in view 7074 in fig. 7P, the layers 7060-1 to 7060-3 each have a respective thickness, and without separation between the layers 7060-1 to 7060-3, the layers 7060-1 to 7060-3 appear to be stacked (e.g., such that the total thickness of the user interface element 7060 is the sum of the thicknesses of each of the layers 7060-1 to 7060-3). In some embodiments, layer 7060-1 casts shadows onto layers 7060-2 and/or 7060-3 according to the thickness of layer 7060-1. Similarly, in some embodiments, layer 7060-2 casts shadows onto layer 7060-1 and/or layer 7060-3 according to the thickness of layer 7060-2. Likewise, in some embodiments, layer 7060-3 casts shadows onto layer 7060-1 and/or layer 7060-2 depending on the thickness of layer 7060-3. In some embodiments, the layers 7060-1 to 7060-3 cast shadows even without separation between the layers (e.g., as shown in fig. 7N and described herein with reference to the figure). In some embodiments, as shown, for example, in view 7076, layers 7060-1 through 7060-3 cast shadows to each other while separate: layer 7060-1 casts a shadow onto layer 7060-2 (indicated by the shadow region of layer 7060-2), which in turn casts a shadow onto layer 7060-3 (indicated by the shadow region of layer 7060-3). In some embodiments, the appearance of the shadow is based on the degree of separation between the layers. For example, as shown in view 7078, layer 7060-1 casts a more diffuse shadow onto layer 7060-2 than the shadow in view 7076 in accordance with an increased degree of separation between layers 7060-1 and 7060-2 in view 7078 relative to the degree of separation in view 7076. Similarly, as shown in view 7078, layer 7060-2 casts a more diffuse shadow onto layer 7060-3 than the shadow in view 7076 in accordance with an increased degree of separation between layers 7060-2 and 7060-3 in view 7078 relative to the degree of separation in view 7076.

In some implementations, the layers of the user interface element project light onto each other. For example, the shaded areas (described herein as shadows) in views 7074, 7076, and 7078 in fig. 7P may alternatively represent light. Thus, similar to the behavior described above with reference to shading, in some embodiments, the light projected onto each other by the layers 7060-1 to 7060-3 has a different appearance based on the degree of separation between the layers. For example, according to the greater degree of separation between the layers 7060-1 to 7060-3 in view 7076 than in view 7078, the light cast by layers 7060-1 and 7060-2 in view 7078 (e.g., represented by shading) is more diffuse than the light cast by layers 7060-1 and 7060-2 in view 7076. In some implementations, light projected by the first layer onto the second layer (or elsewhere in the environment 7104) is displayed so as to simulate light emitted by the first layer. In some embodiments, light projected by the first layer onto the second layer (or elsewhere in the environment 7104) is displayed so as to simulate light reflected off the first layer from one or more individual light sources (e.g., one or more physical and/or simulated light sources) in the environment 7104.

In some embodiments, the icon 7060 is displayed with different amounts of separation between the layers 7060-1 through 7060-3 based on different types of user inputs directed to the icon 7060 and/or different degrees to which the user inputs have been made. In scene 7062 in fig. 7P, the user 7002 does not interact with the icon 7060. In scene 7064 in fig. 7P, the user 7002 is looking at the icon 7060. In the scenario 7066 in fig. 7P, the user 7002 is looking at the icon 7060, and the user's hand 7020 is in a ready state (e.g., a state in which the user's hand and/or finger position indicates that the user is ready to perform an air gesture (such as pinching while the user's gaze is directed at the icon, or tapping at a position corresponding to the icon in a three-dimensional environment), with the hand 7020 lifted toward the computer system 101, in a configuration in which the index finger protrudes toward the computer system 101. Although a user's left hand 7020 is shown in fig. 7P, one of ordinary skill in the art will recognize that the user's right hand 7022 may additionally or alternatively be used to provide input.

In some embodiments, in the absence of user interaction with the icon 7060, as shown in the scene 7062, the icon 7060 is displayed without any separation between the layers 7060-1 to 7060-3, as shown in the view 7074. In some implementations, as shown in scene 7064, in response to the user 7002 focusing on the icon 7060 based on gaze, the icon 7060 is visually distinguished in the environment 7104, and in particular, slightly scaled up relative to other icons in the group of icons 7070, and is sketched to be displayed as casting a shadow behind it in the environment 7104 (e.g., on other icons in the group of icons 7070). In some implementations, the layers 7060-1 through 7060-3 of the icon 7060 are separated to a first degree shown in view 7076 as shown in scene 7064 in response to the user 7002 focusing on the icon 7060 based on the gaze.

In some implementations, as shown in scene 7066, in response to the user 7002 focusing on the icon 7060 based on gaze and hand status (e.g., hand position and configuration), the icon 7060 continues to be visually distinguished relative to other icons in the group of icons 7070 in the environment 7104 (e.g., slightly scaled up, outlined, and/or shadowed relative to other icons in the group of icons 7070). Optionally, the type of visual distinction between icons that are the subject of the user's attention and other icons changes as the user input proceeds (e.g., as the user 7002 brings the hand 7020 into a ready state while looking at the icon 7060, or as the user 7002 shifts gaze to the icon 7060 while the hand 7020 is in a ready state), such as by changing the scale of the icon 7060, changing the outline of the icon 7060, and/or changing the simulated shadow cast by the icon 7060 into the environment 7104. In some implementations, as shown in the scene 7066, in response to the user 7002 focusing attention on the icon 7060 based on gaze and hand status, the layers 7060-1 to 7060-3 of the icon 7060 are separated to a second degree shown in the view 7078.

In some implementations, in a transition from the scene 7066, when the icon 7060 is displayed with a degree of separation indicated in the view 7078, and when the gaze of the user 7002 is directed at 7060 and the hand 7020 of the user 7002 is in a ready state, the user 7002 performs a selection gesture to select the icon 7060 (e.g., an air gesture such as a pinch when the gaze of the user is directed at the icon, or a tap at a location corresponding to the icon in a three-dimensional environment). In some embodiments, the selection gesture is a pinch gesture that includes two or more fingers (e.g., thumb and index finger) of the hand moving toward each other. In some implementations, as the selection gesture progresses (e.g., two fingers move toward each other before contacting), the amount of separation between the layers 7060-1 to 7060-3 decreases (e.g., the icon 7060 transitions from or at least partially transitions from an appearance having the degree of separation shown in view 7078 to an appearance having the degree of separation shown in view 7076). In some implementations, in conjunction with decreasing the degree of separation in response to the progression of the selection gesture, the icon 7060 is changed in scale (e.g., decreased in scale). In some implementations, if the progress of the selection gesture is reversed (e.g., two fingers move away from each other before contacting), the transformation of the icon 7060 is reversed: for example, the amount of separation between the layers 7060-1 through 7060-3 increases (e.g., the icon 7060 transitions from an appearance having the degree of separation shown in view 7076 or at least partially transitions to an appearance having the degree of separation shown in view 7078) and/or the icon 7060 is scaled (e.g., scaled up).

In some implementations, as the selection gesture is completed (e.g., two fingers contact and/or separate after contact), optionally while the gaze of the user 7002 remains on the icon 7060, the amount of separation between the layers 7060-1 to 7060-3 increases again and/or the icon 7060 is changed in scale (e.g., increased in scale). In some implementations, the transformation of the icon 7060 in response to completion of the selection gesture at least partially reverses the transformation of the icon 7060 in response to initiation of the selection gesture (e.g., initial movement of two fingers toward each other prior to contact). In some implementations, in response to completion of the selection gesture, an operation associated with the icon 7060 is performed in the environment 7104. For example, an application (such as an email application associated with icon 7060) is launched and a user interface associated with the application (such as an email application user interface) is displayed (e.g., an application tray having the characteristics and behavior of user interface elements 7030, 7040, and/or 7054 described herein with reference to fig. 7A-7O). In another example, the folder is opened and its contents are displayed. In another example, a communication session is initiated.

In some implementations, the transformation of the icon 7060 in response to the user 7002 focusing on the icon 7060 includes animation of one or more elements of the icon 7060 (e.g., in addition to separation of the layers 7060-1 through 7060-3 and/or rescaling of the icon 7060). For example, the envelope graphics are optionally animated to show that the envelope is opening (e.g., by opening the top cover of the envelope), shaking or vibrating the envelope, or other animation. In some implementations, the icon 7060 is displayed or animated in response to the user 7002 performing a selection gesture directed to the icon 7060.

Fig. 7Q-7W illustrate changing an appearance of a selectable user interface object in response to an input corresponding to the selectable user interface object and indicating that interaction with the selectable user interface object is ready, according to some embodiments. The user interfaces in fig. 7Q to 7W are used to illustrate the processes described below, including the process in fig. 13. According to some embodiments, the user interface elements described with reference to fig. 7Q-7W optionally have one or more of the characteristics and/or behaviors of the user interface elements described herein with respect to other examples (such as examples in fig. 7A-7G, 7H-7J, 7K-7O, 7P, and/or 7X-7Z), unless otherwise indicated in the specification.

As shown in the example in fig. 7Q to 7W, contents visible via the display generating section 7100 of the computer system 101 are displayed on a touch screen held by the user 7002. In some embodiments, the display generating component 7100 of computer system 101 is a head-mounted display worn on the head of user 7002 (e.g., the content shown in fig. 7Q-7W as visible via display generating component 7100 of computer system 101 corresponds to the field of view of user 7002 when wearing the head-mounted display).

Fig. 7Q illustrates a view of an environment 7104 that is visible via a display generating component 7100 (e.g., having one or more portions displayed via the display generating component and/or one or more passthrough portions visible through a transparent or translucent display). The view of environment 7104 includes a view of wall hanging 7106, a view of virtual ball 7108, and a view of box 7014'. Fig. 7Q also shows a user interface element 7080 displayed over a view of environment 7104. The user interface element 7080 may be a user interface of an application executing on the computer system 101, a system user interface (e.g., a setup user interface or an application launch user interface), an alert or notification, or a setup menu. The user interface element 7080 is displayed as a three-dimensional object having an appearance made of a simulated material (e.g., a pane or tray of simulated material, such as a transparent or translucent material) with a thickness that is visible along the top and left edges of the user interface element 7080 from the user's point of view in fig. 7Q. Further, depending on whether the user interface element 7080 is displayed as a transparent or partially transparent material, the thickness of the user interface element 7080 is clearly visible from the right and bottom edges from the viewpoint of the user in fig. 7Q. The user interface element 7080 has an appearance based on the underlying portion of the environment 7104 (such as wall 7004', wall hanging 7106, ball 7108, and/or box 7014'), an appearance that optionally uses graphical composition of frames from different times, color gradients, or other exemplary techniques described herein. Similar to the user interface elements described herein with reference to fig. 7A-7J, in some embodiments, the appearance of the user interface element 7080 is also updated over time based on changes in the appearance of overlapping portions of the environment 7104.

The user interface element 7080 includes a plurality of regions. Specifically, the user interface element 7080 includes a first region 7080-a (e.g., a pane or tray of simulated material) that defines a front surface (e.g., a front face or front plane) of the user interface element 7080. In some embodiments, the region 7080-a of the user interface element 7080 is similar to the background portion 7030-a of the user interface element 7030 (fig. 7D), the background portion 7040-a of the user interface element 7040 (fig. 7H), and/or the first region 7054-a of the user interface element 7054 (fig. 7L). The user interface element 7080 also includes regions 7080-b, 7080-c, and 7080-d that correspond to selectable objects, such as buttons, menus, or other controls, that can be activated to perform respective associated operations in the environment 7104. As shown in fig. 7Q, the regions 7080-b, 7080-c, and 7080-d are flat and flush with the surface of the user interface element 7080 (e.g., when the user 7002 does not indicate that it is ready to interact with any of the regions 7080-b, 7080-c, and 7080-d, such as by not focusing attention on any of these regions), as indicated in a top view 7082, which shows a top view of the user interface element 7080 and its regions 7080-b, 7080-c, and 7080-d. The appearances of the regions 7080-b, 7080-c, and 7080-d in fig. 7Q are optionally default appearances (e.g., default settings or values based on one or more visual properties, such as brightness or darkness, opacity or transparency, size, thickness, amount of specular reflection, degree of blurring, and/or degree of separation from the user interface element 7080) that indicate that the regions 7080-b, 7080-c, and 7080-d are not currently selected for further interaction, and optionally indicate that the regions 7080-b, 7080-c, and 7080-d are in a first (e.g., closed or inactive) state.

The user interface element 7080 is shown as a material having optical properties that interact with light in the environment 7104. As shown in fig. 7Q, the user interface element 7080 is displayed with an appearance (e.g., based on visual effects) that reflects light from one or more sources (e.g., one or more physical light sources in a physical environment surrounding the computer system 101 and/or one or more simulated light sources in the environment 7104, or a subset thereof) as it appears to a person viewing the user interface element 7080 at a location in the environment 7104 in fig. 7Q and from the user's point of view in fig. 7Q. For example, depending on the light source above and to the left of the user interface element 7080, fig. 7Q shows specular reflection at the upper left corner of the user interface element 7080 (e.g., at the upper left corner of the top edge) (e.g., at a location along the outer edge of the user interface element 7080 facing a given light source) and specular reflection at the lower right inner corner of the user interface element 7080 (e.g., internal specular reflection is shown along the inner side of the edge of the user interface element 7080 opposite the location of the external specular reflection in the upper left corner, wherein the amount of internal specular reflection at the lower right corner is less than the amount of external specular reflection at the upper left corner).

Fig. 7R illustrates a transition made in the user interface element 7080 in response to the user 7002 indicating that it is ready to interact with the user interface element 7080. Fig. 7R illustrates that the user 7002 is focusing (e.g., gazing) on the region 7080-b, which is indicated in fig. 7R by a dashed line extending from the eyes of the user 7002 to the region 7080-b. Before receiving an interaction input directed to the region 7080-b, the user 7002 focuses on the region 7080-b indicating that the user 7002 is ready to interact with the region 7080-b. In some embodiments, in order for the input to indicate that the user 7002 is ready to interact with a corresponding region (such as region 7080-b), the hand of the user 7002 (e.g., left hand 7020, or right hand 7022 if the computer system 101 is not held in the right hand 7022) must be in a ready state (e.g., as an alternative or complement to the requirement that the user 7002 focus on the corresponding region).

As shown in fig. 7R, in response to the user 7002 focusing on the region 7080-b, the visual appearance of the region 7080-b is changed (e.g., to an appearance that indicates that the region 7080-b is selected for further interaction): the thickness of region 7080-b increases; the opacity of the region 7080-b decreases (making the region 7080-b more transparent); and the region 7080-b is lifted off the surface of the region 7080-a and more generally away from the surface of the user interface element 7080 so as to appear to hover over the surface of the user interface element 7080 (e.g., separate from the surface of the user interface element 7080, such as in the z-direction of an x-y-z coordinate system in which the front surface of the user interface element 7080 defines an x-y plane) and moved closer to the viewpoint of the user 7002, as shown in environment 7104 and indicated in top view 7082 in fig. 7R. Thus, shadows are displayed as cast by the raised region 7080-b onto the surface of the user interface element 7080 (e.g., due to an increase in the thickness of the region 7080-b and/or a separation of the region 7080-b from the surface of the user interface element 7080). In some embodiments, the increase in thickness of the region 7080-b is displayed via an animated transition, such as an animation of the region 7080-b gradually increasing in thickness. Optionally, other visual changes are also animated. In some embodiments, the size of the region 7080-b is maintained (e.g., in the x-dimension and the y-dimension) in response to the user 7002 focusing on the region 7080-b. In some embodiments, the brightness of region 7080-b increases.

In some embodiments, as shown in fig. 7R, the region 7080-b is displayed as a material having optical properties that interact with light in the environment 7104, and according to an increase in the thickness of the region 7080-b, the elevated region 7080-b is displayed as appearing to reflect light as indicated by specular reflection from the upper left corner of the elevated region 7080-b (e.g., consistent with light sources corresponding to other specular reflections displayed on the user interface element 7080 in fig. 7R, as described herein with reference to fig. 7Q). In some embodiments, the amount of specular reflection and/or the degree to which specular reflection extends along one or more edges of the region 7080-b is indicative of the magnitude of the thickness of the region 7080-b (e.g., the amount and/or degree of specular reflection from the region 7080-b gradually increases or decreases (e.g., during animation) as the thickness of the region 7080-b gradually increases or decreases, respectively). In some embodiments, as shown in fig. 7R, in response to the user 7002 focusing on the region 7080-b, the amount of blurring of the region 7080-b increases as the thickness of the region 7080-b increases and is lifted off the user interface 7080 (e.g., to enhance the visual effect that translucent material is lifted off of the underlying object, which would make the underlying object appear more blurred through the lifted-off material).

The appearance of the regions 7080-c and 7080-d remain unchanged in fig. 7R, according to the user 7002 not indicating that it is ready to interact with those regions (e.g., not focusing attention on those regions, such as by looking at those regions).

In some cases, environment 7104 includes a user interface element that includes a slider (e.g., for selecting a particular set value from a set or range of values, such as an audio volume level or a brightness level, or for selecting a location within content, such as a playback location within audio and/or video). For example, the user interface element 7080 optionally also includes a slider. In some implementations, the slider is displayed without a control (e.g., slider thumb or arrow button) for changing the current value of the slider being displayed before entering an intent indicating the user 7002 to interact with the slider. In some implementations, a control for changing the current value of the slider is displayed in response to an input indicating an intent of the user 7002 to interact with the slider. In some embodiments, the display of the slider control requires that the user 7002 be focused on the slider and that the user's 7002 hand be in a ready state (while in some embodiments only one of the two conditions is sufficient to trigger the display of the slider control). In some embodiments, the displayed slider control exhibits some of the same behavior as the lift region 7080-b described above with reference to fig. 7R and/or has one or more of the visual attributes of the lift region. For example, in some embodiments, the slider control has a non-zero thickness and/or a non-zero degree of separation from the user interface element of which the slider is a part. Optionally, the slider control has an opacity corresponding (e.g., equal or proportional) to the opacity of the lift region 7080-b. Optionally, the slider and/or slider control exhibit specular reflection, while in some embodiments one or both of the slider or slider control does not exhibit specular reflection (e.g., unlike buttons and other control areas). Optionally, the slider control casts shadows onto underlying content, such as the slider and/or surrounding user interface elements. In some embodiments, the display of the slider control is stopped if the user 7002 is stopped focusing on the slider, or if the user's 7002 hand is stopped in a ready state, or in some embodiments, if both of these occur.

In some cases, environment 7104 includes a user interface element that includes a toggle switch (e.g., for selecting one of a plurality (typically two) mutually exclusive values of a particular setting, such as for enabling or disabling a function or feature, or toggling between two modes). For example, the user interface element 7080 optionally also includes a toggle switch. The switch typically includes a movable element (e.g., a disc or other shape) superimposed over a background element, different regions of the background element representing different values of the setting (e.g., a bar or groove, one end representing one value and the opposite end representing the other value). In some implementations, in a default appearance of the toggle switch (e.g., displayed when the user 7002 is not focused on the toggle switch and/or when the user's 7002 hand is not in a ready state), the background element is a recessed area (e.g., similar to recessed area 7054-c (fig. 7M)) and has a corresponding depth relative to a front surface of the surrounding user interface element. In some implementations, the movable element is positioned in the recessed background region. In some embodiments, the movable element exhibits some or all of the same behavior as the boost region 7080-b described above with reference to fig. 7R and/or has one or more of the visual attributes of the boost region (e.g., reduced opacity, increased brightness, maintained size, and/or projection of shadows onto underlying content, such as onto background elements). In some embodiments, in a default state, the movable element has a respective thickness and/or height relative to the recessed background region. In some such embodiments, in response to the user 7002 focusing on the toggle switch, the thickness of the movable element does not increase (e.g., because the movable element already has a non-zero thickness) and/or is not lifted off or otherwise changes height relative to the recessed background area, so the shadow cast onto the underlying content is not enhanced, but the movable element optionally exhibits other behavior similar to that of the area 7080-b (fig. 7R), such as maintaining the same size, changing opacity, and/or changing brightness.

Fig. 7S illustrates a transition made in the user interface element 7080 in response to the user 7002 indicating that the user interface element 7080 is ready to interact with a different portion of the user interface element 7080 than in fig. 7R. Fig. 7S shows that the user 7002 is focusing (e.g., looking) at the region 7080-c, which is indicated in fig. 7S by a dashed line extending from the eyes of the user 7002 to the region 7080-c instead of extending to the region 7080-b as in fig. 7R. Focusing the attention of the user 7002 to the region 7080-c before receiving an interaction input directed to the region 7080-c indicates that the user 7002 is ready to interact with the region 7080-c. As shown in fig. 7S, in response to the user 7002 focusing on the region 7080-c, the visual appearance of the region 7080-c is changed to indicate the appearance of the region 7080-c selected for further interaction (and with respect to some visual attribute, such as size, not changed in a manner similar to that previously described herein with reference to the region 7080-b (fig. 7R)) in a manner similar to that previously described herein with reference to the region 7080-b (fig. 7R). For example, the thickness of the region 7080-c increases, the opacity of the region 7080-c decreases (such that the region 7080-c becomes more transparent), and the region 7080-c is lifted off the surface of the user interface element 7080 so as to appear to hover over the surface of the user interface element 7080 and closer to the viewpoint of the user 7002, as shown in environment 7104 and indicated in the top view 7082 in fig. 7S. Thus, shadows are displayed as cast by the lift-off region 7080-c onto the surface of the user interface element 7080. In some embodiments, the brightness of the region 7080-c increases. In some embodiments, similar to region 7080-b, region 7080-c is shown as a material having optical properties that interact with light in environment 7104, and according to an increase in the thickness of region 7080-c, elevated region 7080-c is shown as appearing to reflect light as indicated by specular reflection from the upper left corner of elevated region 7080-c (e.g., consistent with the same light sources in environment 7104 as described herein with reference to fig. 7Q-7R). When viewed through the boost region 7080-c, the ambiguity of the underlying content (such as region 7080-a of the user interface element 7080) also increases.

In addition, in response to the user 7002 no longer focusing on the region 7080-b, the visual appearance of the region 7080-b reverts to its appearance as shown in FIG. 7Q and described with reference to that figure (e.g., a default appearance indicating that the region 7080-b is not currently selected for further interaction). For example, the region 7080-b is reduced in thickness (e.g., restored to flat) and thus no longer displayed as having specular reflection, increased in opacity, reduced in the amount of blurring of underlying content (such as the region 7080-a), and replaced on the surface of the user interface element 7080 (e.g., restored to flush with the surface of the user interface element 7080), and thus no longer shown as casting shadows onto the user interface element 7080, as shown in environment 7104 and indicated in top view 7082 in fig. 7S. In some embodiments, the brightness of region 7080-b is reduced. The appearance of the region 7080-d remains unchanged in FIG. 7S in response to the user 7002 not indicating preparation to interact with the region 7080-d.

Fig. 7T illustrates a transition in the user interface element 7080 in response to the user 7002 focusing (e.g., looking) at the region 7080-b (e.g., again, as is the case in the scene of fig. 7R), which is indicated in fig. 7T by a dashed line extending from the eyes of the user 7002 to the region 7080-b instead of extending to the region 7080-c as in fig. 7S. Directing the attention of the user 7002 to the region 7080-b indicates that the user 7002 is ready to interact (e.g., again) with the region 7080-b. As shown in fig. 7T, in response to the user 7002 focusing on the region 7080-b, the visual appearance of the region 7080-b is changed to the same appearance as in fig. 7R in a manner similar to that previously described herein with reference to the region 7080-b of fig. 7R and the region 7080-c of fig. 7S, thereby indicating that the region 7080-b is selected (e.g., reselected) for further interaction: the thickness of region 7080-b increases; the opacity of region 7080-b decreases; the blurring degree of the underlying content increases; and the region 7080-b is lifted off the surface of the user interface element 7080 so as to appear to hover over the surface of the user interface element 7080 and closer to the viewpoint of the user 7002, as shown in environment 7104 and indicated in top view 7082 in fig. 7T. Thus, the shadow is displayed as projected by the lift region 7080-b onto the surface of the user interface element 7080, and the lift region 7080-b is displayed with specular reflection from the upper left corner of the lift region 7080-b and optionally the brightness of the region 7080-b increases.

In addition, in response to the user 7002 no longer focusing on the region 7080-c, the visual appearance of the region 7080-c reverts to its appearance as shown in fig. 7Q and described with reference to that figure (e.g., a default appearance indicating that the region 7080-c is not currently selected for further interaction) via a change similar to the change to the region 7080-b in fig. 7S. For example, the region 7080-c is reduced in thickness (e.g., flattened) and thus no longer displayed as having specular reflection, increased opacity, reduced amount of blurring of underlying content, and replaced on the surface of the user interface element 7080 (e.g., so that the surface is flush), and thus no longer shown as casting shadows onto the user interface element 7080, as shown in environment 7104 and indicated in top view 7082 in fig. 7R, and optionally reduced in brightness (e.g., restored). The appearance of the region 7080-d remains unchanged in FIG. 7T in response to the user 7002 not indicating that it is ready to interact with the region 7080-d.

Fig. 7U illustrates a transition from fig. 7T (or from fig. 7R) based on the user 7002 performing an input gesture with his hand 7020 (e.g., an air gesture such as a pinch or tap at a location corresponding to the region 7080-b in a three-dimensional environment), as indicated by the arrowed hand 7020 shown alongside the hand 7020 in fig. 7U. In some implementations, the input gesture is initiated when the user 7002's attention is focused on the user interface element 7080 (e.g., specifically on the region 7080-b). In some implementations, the input gesture is initiated after the hand 7020 is brought into the ready state (e.g., the input gesture is initiated from the ready state). While fig. 7U shows the left hand 7020 of the user 7002, one of ordinary skill in the art will recognize that the right hand 7022 of the user may additionally or alternatively be used to provide input. In fig. 7U, in response to the user 7002 focusing on the region 7080-b and performing an input gesture, thus interacting with the region 7080-b, the visual appearance of the region 7080-b changes from its appearance in fig. 7T (e.g., further changes from its appearance in fig. 7Q), as shown in environment 7104 and indicated in top view 7082. For example, the size of the lift region 7080-b is reduced, the opacity of the lift region 7080-b is reduced (e.g., further), and the lift region 7080-b is darkened (e.g., the brightness is optionally reduced to a brightness that is less than the default brightness as shown in fig. 7Q). In addition, shadows that are displayed as being cast by the lift region 7080-b onto the surface of the user interface element 7080 are enhanced (e.g., lengthened and/or darkened). Optionally, one or more visual properties of the elevated region 7080-b are maintained in response to an input gesture, such as thickness, corresponding specular reflection, an amount of blurring of underlying content, and/or separation from a surface of the user interface element 7080, as also shown in environment 7104 and indicated in top view 7082 in fig. 7U. Based on the absence of inputs directed to those regions (e.g., neither any input indicating ready interaction nor any input actually interacting with those regions is detected), the appearances of regions 7080-c and 7080-d are maintained throughout fig. 7U-7W.

As described above, in some cases, the respective user interface element in environment 7104 includes a slider. In some embodiments, the control for changing the current value of the slider, when displayed with an appearance that is responsive to the user 7002 indicating an intent to interact with the slider, exhibits some of the same behaviors as the lift region 7080-b described above with reference to fig. 7U and/or has one or more of the visual attributes of the lift region (e.g., responsive to an input gesture of the hand 7020). For example, in response to an input gesture similar to the input gesture of fig. 7U, the slider control is reduced in size, reduced in opacity, and/or darkened (while in some embodiments, the opacity and/or brightness is maintained). Optionally, shadows cast by the slider control onto the underlying content are enhanced. In some embodiments, one or more of the visual attributes of the slider control are maintained, such as thickness, corresponding specular reflection (if any), amount of blurring of underlying content, and/or amount of separation (if any) from the user interface element of which the slider is a part. The value indicated by the slider optionally changes during the input gesture in response to movement of the input gesture (e.g., in response to lateral movement of the hand 7020 while in the pinch-in-air or flick-in-air configuration), and the amount of change in the value of the slider is optionally based on the amount of movement of the input gesture (e.g., proportional to or accelerated with the amount of movement of the input gesture).

As described above, in some embodiments, the respective user interface elements in environment 7104 include a toggle switch. In some embodiments, the movable element (e.g., a disc) of the toggle switch, when displayed as having an appearance that is responsive to the user 7002 indicating an intent to interact with the toggle switch, exhibits some of the same behaviors as the lift region 7080-b described above with reference to fig. 7U and/or has one or more of the visual attributes of the lift region (e.g., responsive to an input gesture of the hand 7020). For example, in response to an input gesture similar to the input gesture of fig. 7U, the movable switching element is reduced in size, reduced in opacity, and/or darkened (while in some embodiments, the opacity and/or brightness is maintained). Optionally, shadows cast by the movable switching element onto the underlying content (including, for example, background elements) are enhanced. In some implementations, one or more of the visual attributes of the movable element are maintained, such as thickness, corresponding specular reflection (if any), amount of blurring of underlying content, and/or amount of separation from surrounding user interface elements (if any).

Fig. 7V illustrates a transition from fig. 7U in response to completion of an input gesture performed by the user 7002 and the attention of the user 7002 remains directed to the region 7080-b. In response to the end of the input gesture, computer system 101 performs a respective operation associated with region 7080-b (e.g., activates a button or control represented by region 7080-b, launches an application, presents a menu, dismisses an alert, switches a function, or other operation) and changes the visual appearance of region 7080-b from its appearance in fig. 7U. Specifically, in fig. 7V, the size of the lift region 7080-b is increased (e.g., restored to the same size as any one of fig. 7Q to 7T, or to a different size, such as a larger size than in fig. 7Q), the opacity of the lift region 7080-b is increased (e.g., restored to the same opacity as in fig. 7T, restored to the same opacity as in fig. 7Q, or to a different level of opacity, such as increased to a greater opacity than in fig. 7Q), and the brightness of the lift region 7080-b is increased (e.g., restored to the same brightness as in fig. 7T, restored to the same brightness as in fig. 7Q, or to a different brightness level, such as increased to a greater brightness than in fig. 7Q). In addition, shadows shown projected by the lift region 7080-b onto the surface of the user interface element 7080 are reduced in intensity (e.g., shortened and/or lightened, e.g., reduced to the same intensity as in fig. 7T or fig. 7Q). Optionally, one or more visual properties of the elevated region 7080-b, such as thickness, specular reflection, separation from the surface of the user interface element 7080, and shadow intensities are maintained as the user 7002's attention remains focused on the elevated region 7080-b after the input gesture ends, as shown in environment 7104 and indicated in top view 7082 in fig. 7V. In some implementations, the respective operations associated with the region 7080-b are performed in response to the end of the input gesture of fig. 7U regardless of (e.g., even if) whether the attention of the user 7002 remains focused on the region 7080-b. In some implementations, the respective operations associated with region 7080-b are performed in response to the end of the input gesture of fig. 7U and the attention of user 7002 has been moved away from region 7080-b (e.g., performance of the respective operations is delayed until the same situation occurs in which the attention of user 7002 is no longer focused on region 7080-b in response to the end of the input gesture).

As described above, in some cases, the environment 7104 includes a user interface element that includes a slider, and in some embodiments, a control for changing a current value of the slider, when displayed, exhibits some of the same behaviors as the lift region 7080-b described above with reference to fig. 7U and/or has one or more of the visual properties of the lift region (e.g., in response to an input gesture of the hand 7020 in fig. 7U). Additionally, in some embodiments, the slider control exhibits some of the same behavior as the lift region 7080-b described above with reference to fig. 7V and/or has one or more of the visual attributes of the lift region (e.g., in response to the end of the input gesture of fig. 7U). For example, in response to the end of an input gesture similar to the input gesture of fig. 7U, in some embodiments the slider control is increased (e.g., restored) in size, increased in opacity (e.g., restored), and/or lightened (e.g., restored to the same brightness as when the user 7002 indicated an intent to interact with the slider control prior to the input gesture) (while in some embodiments, the opacity and/or brightness is maintained). Optionally, shadows cast by the slider control onto the underlying content are reduced in intensity (e.g., to the same intensity as when the user 7002 indicated the intent to interact with the slider control, or to the same intensity as before the user 7002 indicated the intent to interact). In some embodiments, the slider control continues to be displayed while the user 7002 remains focused on the slider or, in particular, on the slider control, while one or more visual properties are maintained, such as thickness, corresponding specular reflection (if any), amount of blurring of underlying content, and/or separation from user interface elements of which the slider is a part, and after the user 7002 is moved away from the slider, the slider control ceases to be displayed (e.g., fades away). In some implementations, the display of the slider control is stopped after the input gesture ends (e.g., regardless and even though the user 7002 is still focused on the slider).

As described above, in some cases, the environment 7104 includes a user interface element including a toggle having a movable element that, in some embodiments, exhibits some of the same behavior as the lift region 7080-b described above with reference to fig. 7U and/or has one or more of the visual attributes of the lift region (e.g., in response to an input gesture of the hand 7020 in fig. 7U). Additionally, in some embodiments, the movable element exhibits some of the same behavior as the lift region 7080-b described above with reference to fig. 7V and/or has one or more of the visual attributes of the lift region (e.g., in response to the end of the input gesture of fig. 7U). For example, in response to the end of an input gesture similar to the input gesture of fig. 7U, the movable element is increased (e.g., restored) in size, increased (e.g., restored) in transparency, and/or lightened (e.g., restored to the same default brightness as before the user 7002 indicated the intent to interact with the toggle switch, restored to the same brightness as in response to the user 7002 indicating the intent to interact with the toggle switch, or increased to a different brightness level, such as increased to a brightness greater than the default brightness) (while in some embodiments, the opacity and/or brightness is maintained). Optionally, the shadow cast by the movable element onto the underlying content is reduced in intensity (e.g., to the same intensity as when the user 7002 indicated the intent to interact with the toggle, or to the same intensity as before the user 7002 indicated the intent to interact). In some implementations, lateral movement of the movable element is displayed in response to completion of the input gesture. For example, the movable element moves from a position corresponding to one switching value relative to the background element of the switch to a different position corresponding to a different switching value relative to the background element of the switch (e.g., from one end of the background element to the opposite end). In addition, the appearance of the toggle switch is changed to indicate a new switch value (e.g., via a change in color, such as a change from gray or white to green, or vice versa, or other visual attribute change). In some embodiments, after the input gesture ends, while the user's 7002 attention remains directed to the toggle switch, the movable element continues to be displayed while one or more visual attributes are maintained (e.g., maintaining an increased brightness indicating the user's 7002 intent to interact with the toggle switch, where the increased brightness is applied to the toggle switch, the appearance of which corresponds to a new switch value), and then after the user's 7002 attention is removed from the toggle switch, the one or more visual attributes are restored (e.g., without the user's intent to interact with the toggle switch, the brightness of the toggle switch is restored to a default brightness, while the toggle switch otherwise maintains the same appearance corresponding to the new switch value).

Fig. 7W illustrates a transition from fig. 7V in response to the attention of the user 7002 being moved away from the region 7080-b (as indicated by the dashed line extending from the eyes of the user 7002 and to the left of the user interface element 7080 in fig. 7W) and/or in response to the input gesture performed by the user 7002 in fig. 7U ending (e.g., completing). In some embodiments, fig. 7W represents a transition from fig. 7V in response to the attention of the user 7002 moving away from the region 7080-b after the input gesture performed by the user 7002 in fig. 7U has ended. In some implementations, fig. 7W represents a transition from fig. 7U in response to the user's 7002 attention moving away from the region 7080-b plus the end of the input gesture performed by the user 7002 in fig. 7U. In some embodiments, fig. 7W represents a transition from fig. 7U in response to the end of an input gesture performed by the user 7002 after the user's 7002 attention has been moved away from the region 7080-b. In some implementations, the appearance of the environment 7104 in fig. 7W represents a transition from fig. 7U in response to the end of an input gesture performed by the user 7002 regardless of whether the user's 7002 attention has been moved away from the region 7080-b (e.g., the transition occurs in response to the end of the input gesture even though the user's 7002 attention remains on the region 7080-b). In fig. 7W, the visual appearance of region 7080-b changes from its appearance in fig. 7V: the thickness of the region 7080-b is reduced (e.g., reverted to the same zero thickness as in fig. 7Q, wherein the region 7080-b is flat) and the separation between the region 7080-b and the surface of the user interface element 7080 is reduced (e.g., reverted to the same zero separation as in fig. 7Q, wherein the region 7080-b is flush with the surface of the user interface element 7080), as shown in environment 7104 and indicated in top view 7082 in fig. 7W. The region 7080-b in fig. 7W is brighter and more opaque (e.g., less transparent) than the region 7080-b in fig. 7Q. In some embodiments, the visual appearance of the region 7080-b in fig. 7W indicates that the region 7080-b is in a second state, such as an on or active state (e.g., has been activated to change a setting of the computer system 101 and/or to perform a corresponding operation on the computer system 101).

Fig. 7X-7Z illustrate visually distinguishing different portions of a computer-generated user interface object whose appearance is based on content in a surrounding three-dimensional environment behind the user interface object by applying different transformations to representations of underlying content, according to some embodiments. The user interfaces in fig. 7X-7Z are used to illustrate the processes described below, including the process in fig. 14. According to some embodiments, the user interface elements described with reference to fig. 7X-7Z optionally have one or more of the characteristics and/or behaviors of the user interface elements described herein with respect to other examples (such as examples in fig. 7A-7G, 7H-7J, 7K-7O, 7P, and/or 7Q-7W), unless otherwise indicated in the specification.

As shown in the example in fig. 7X to 7Z, contents visible via the display generating section 7100 of the computer system 101 are displayed on a touch screen held by the user 7002. In some embodiments, the display generating component 7100 of computer system 101 is a head-mounted display worn on the head of user 7002 (e.g., the content shown in fig. 7X-7Z as being visible via the display generating component 7100 of computer system 101 corresponds to the field of view of user 7002 when the head-mounted display is worn).

Fig. 7X shows the same scenario as in fig. 7D, in which a view of environment 7104 at time t=t ₂ is visible via display 7100 of computer system 101. The user interface element 7030 at time t ₂ has an appearance that is based on the appearance of one or more portions of the environment 7104 that are underneath the user interface element 7030. While fig. 7X-7Z illustrate examples of user interface elements 7030 based on fig. 7D-7G whose appearance is based on graphical composition of the appearance of the underlying portion of environment 7104 at one or more times as described herein with reference to fig. 7G, the concepts described herein with reference to fig. 7X-7Z are similarly applicable to user interface elements 7040 of fig. 7H-7J whose appearance is based on color gradients generated from colors sampled from the underlying portion of environment 7104. The graphical composition of fig. 7G and the color gradients of fig. 7H-7J are both examples of what is referred to herein as a blurred representation of the underlying portion of environment 7104.

Fig. 7X also shows an expanded view 7084 of the user interface element 7030. The expanded view 7084 shows four regions that are part of the user interface element 7030. The region 7084-a is a background portion of the user interface 7030 that corresponds to, for example, a non-interactive region such as a region that does not include selectable content (such as text, images, buttons, and/or other controls) (e.g., similar to region 7030-a (fig. 7D), region 7040-a (fig. 7H), and/or region 7054-a (fig. 7L)). Region 7084-b represents an interactive (e.g., selectable) object such as a button, toggle, slider, and/or other control (e.g., similar to regions 7030-b and 7030-c (fig. 7D), regions 7040-b and 7040-c (fig. 7H), and/or regions 7054-D (fig. 7L)). Region 7084-c corresponds to the TEXT "TEXT1" and covers the region occupied by the characters of "TEXT 1". The region 7084-d corresponds to the TEXT "TEXT2" and covers the region occupied by the character "TEXT 2". In some embodiments, as in the example of FIG. 7X, the background region 7084-a, the control region 7084-b, the text region 7084-c, and the text region 7084-d are mutually exclusive regions (e.g., mutually exclusive regions that encompass the user interface element 7030).

As shown in fig. 7X, the appearance of the user interface element 7030 is the result of different visual transformations applied to different regions of the starting point representation (also referred to as a blurred representation) of the one or more underlying portions of the environment 7104. In some embodiments, the starting point representation is a composition derived from the graphical composition in scene 7036 in fig. 7G at time t ₂, the appearance of user interface elements 7030-d being based on the resulting composition. In some embodiments, the starting point representation is a generated color gradient, and the appearance of the user interface element 7040 at time t ₂ in fig. 7H is based on the generated color gradient. in some embodiments, the color saturation in the starting point representation (e.g., at any given time, including at time t ₂ and/or at time t ₃) is increased relative to the appearance of the underlying portion of the environment 7104 upon which the starting point representation is based (e.g., generating a composition or color gradient includes increasing the color saturation, or an additional step of increasing the color saturation is applied to the generated composition or color gradient to produce the starting point representation). In fig. 7X, the appearance of the background region 7084-a of the user interface element 7084 is selectively applied to a region represented by a starting point corresponding to a region included in the background region 7084-a by a first transformation, referred to herein as a "transformation X _A" (e.g., as if an image mask in the shape of the applied background region 7084-a was represented to the starting point prior to application of the transformation X _A (e.g., excluding other regions)). In some implementations, transform X _A darkens (e.g., reduces the brightness of) the region to which transform X _A is applied. In some embodiments, transform X _A applies a gray filter (e.g., to darken the region to which transform X _A is applied, increase the gray level of the region, and/or decrease the color saturation of the region (e.g., to offset the increased color saturation represented by the starting point or to restore the color saturation represented by the starting point)). A larger view of the effect of transformation X _A is shown in fig. 7Z, where panel 7088-a shows the result of transformation X _A applied to the entire starting point representation of time t ₂. The transformation X _A is not applied to some other region of the user interface element 7084. in the example of FIG. 7X, the transform X _A is not applied to any of the regions 7084-b, 7084-c, and 7084-d. Instead, different transformations are applied to those regions.

For example, the appearance of the control region 7084-b is generated by selectively applying a different, second transform, referred to herein as "transform X _B," to the region represented by the starting point corresponding to the region included in the control region 7084-b (e.g., as if an image mask (e.g., two rectangles) in the shape of the control region 7084-b were applied to the starting point representation prior to application of transform X _B), and in some embodiments, transform X _B lightens (e.g., increases the brightness of) the region to which transform X _B was applied. A larger view of the effect of transform X _B is shown in fig. 7Z, where panel 7088-b shows the result of transform X _B applied to the entire starting point representation of time t ₂. In the example of FIG. 7X, the transform X _B is not applied to any of the regions 7084-a, 7084-c, and 7084-d.

Regions 7084-c and 7084-d of user interface element 7084 in FIG. 7X illustrate two different transformation examples applied to text regions. For example, the appearance of text region 7084-c is the result of: the forgoing application of the transform X _A described with respect to the background region 7084-a, the forgoing application of the transform X _B described with respect to the control region 7084-b, and the forgoing application of the transform X _D described below with respect to the other text regions 7084-d (e.g., No transformation is applied to the starting point representation, or a visual transformation that is an identity transformation is applied to the starting point representation). In other words, the appearance of the text region 7084-c is a result of maintaining the appearance of the start point transformation (e.g., maintaining the graphical composition or color gradient). A larger view of the effect of giving up applying any transform to the starting point representation of time t ₂ (e.g., a larger view of the appearance of the starting point representation of time t ₂) is shown in fig. 7Z, where panel 7088-c shows the starting point representation without any transform applied. refraining from applying some or all of the one or more visual transformations to the text character region within the corresponding region to which the one or more visual transformations are otherwise applied is a way to improve legibility of text (e.g., non-transformed regions). In some embodiments, the appearance of the text region 7084-c is the result of: a fourth transform referred to herein as "transform X _C" is applied (transform X _C is different from transforms X _A, X _B and X _D), Without applying any of transforms X _A, X _B, and X _D to text region 7084-c, and optionally without applying transform X _C to region 7084-a, 7084-b and 7084-d.

In contrast, the appearance of the text region 7084-d is the result of: a third transformation, referred to herein as "transformation X _D", is selectively applied to the region represented by the starting point at time t ₂ corresponding to the text character region included in text region 7084-d (e.g., as if an image mask in the shape of the characters of text region 7084-d were applied to the starting point representation prior to application of transformation X _D) (e.g., And forgo application of the transform X _A described with respect to the background region 7084-a and forgo application of the transform X _B described with respect to the control region 7084-b), and in some embodiments, the transform X _D darkens the region to which the transform X _D was applied (e.g., Reduce its brightness) to a greater extent than transform X _A darkens the area to which transform X _A was applied. A larger view of the effect of transformation X _D is shown in fig. 7Z, where panel 7088-d shows the result of transformation X _D applied to the entire starting point representation of time t ₂. In the example of FIG. 7X, the transform X _D is not applied to any of the regions 7084-a, 7084-b, and 7084-c. Those of ordinary skill in the art will recognize that the appearance of the text region 7084-d may similarly be generated by applying the transformation X _A to the text region 7084-d (e.g., rather than forgoing application of the transformation X _A) and applying the transformation X _D' that further darkens the text region 7084-d (e.g., Even though X _D' alone does not darken the region more than the transform X _A darkens the region). Applying a visual transformation that darkens (e.g., further darkens) text character regions within a respective region relative to the rest of the respective region is another way to improve legibility of text (e.g., legibility of darkened regions, or in some embodiments, legibility of further darkened regions).

Fig. 7Y shows the same scenario as in fig. 7E, in which, in a transition from the view of environment 7104 at time t=t ₂ as shown in fig. 7X, the view of environment 7104 at time t=t ₃ is visible via display 7100 of computer system 101. The user interface element 7030 at time t ₃ has a changed appearance (fig. 7X) relative to the user interface element 7030 at time t ₂ based on the appearance of one or more changed portions of the environment 7104 under the user interface element 7030, according to the graphical composition of the appearance of the underlying portion of the environment 7104 before time t ₃ and/or including one or more times of time t ₃, as described herein with reference to fig. 7G (or similarly, according to a color gradient generated from a color sampled from the portion of the environment 7104 under the user interface element 7040 at time t ₃, as described herein with reference to fig. 7I).

In some cases, the change in the one or more underlying portions of the environment 7104 is due, at least in part, to movement of other objects in the environment 7104 (e.g., the box 7012 falls into space and/or the ball 7108 scrolls to the right (fig. 7B-7E)). In some cases, the change in one or more underlying portions of the environment 7104 is due, at least in part, to a change in real and/or virtual lighting in the environment 7104 (e.g., as described herein with reference to fig. 7K-7M). In some cases, the change in the one or more underlying portions of the environment 7104 is due, at least in part, to a change in the position of the user interface element 7030 relative to the environment 7104 (e.g., such that the user interface element 7030 is overlaid on a different portion of the environment 7104, such as in the transition from fig. 7D to fig. 7E, wherein the user interface element 7030 rotates relative to the environment 7104 (or in a similar transition from fig. 7H to fig. 7I, wherein the user interface element 7040 rotates relative to the environment 7104) in response to a viewpoint rotation of the user 7002 (e.g., as the user moves a touchscreen or turns their head while wearing a head-mounted display), or such as in a transition similar to the transition from fig. 7I to fig. 7J, wherein the user interface element 7040 moves laterally relative to the environment 7104, such as in response to a drag or other input by the user 7002 to reposition the user interface element 7040 in the environment 7104). In some cases, the change in the one or more underlying portions of the environment 7104 is due, at least in part, to a change in the viewpoint of the user relative to the environment 7104 (e.g., as in the transition from fig. 7D to fig. 7E, wherein the user interface element 7030 rotates relative to the environment 7104 according to the viewpoint of the user 7002 being rotated, as previously described, and/or due to movement of the viewpoint of the user 7002 relative to the user interface element 7030 in the environment 7104 when the user interface element 7030 is world locked (e.g., as if the user moved the touch screen or moved the head of the user interface element 7030 relative to a simulated position of the user interface element 7030 in the environment 7104 while wearing the head-mounted display), thereby changing which portions of the environment 7104 are underneath the user interface element 7030 even without moving the user interface element 7030 relative to the environment 7104). In some cases, the change in the one or more underlying portions of environment 7104 is due to any combination of the above.

Fig. 7Y also shows an expanded view 7086 of the user interface element 7030. Similar to the expanded view 7084 in fig. 7X, the expanded view 7086 shows four regions that are part of the user interface element 7030: region 7086-a is the background portion of user interface 7030 and is similar to region 7084-a in fig. 7X; region 7086-b represents an interactive (e.g., selectable) object and is similar to region 7084-b in fig. 7X; region 7086-c corresponds to TEXT "TEXT1", encompasses the region occupied by the character "TEXT1", and is similar to region 7084-c in fig. 7X; and region 7086-d corresponds to TEXT "TEXT2", covering the region occupied by the character "TEXT2", and is similar to region 7084-d in fig. 7X. In some embodiments, as in the example of FIG. 7Y and similar to the example of FIG. 7X, the background region 7086-a, the control region 7086-b, the text region 7086-c, and the text region 7086-d are mutually exclusive regions (e.g., mutually exclusive regions that encompass user interface elements 7030).

As shown in fig. 7Y, the appearance of the user interface element 7030 is the result of different visual transformations applied to different regions of the starting point representation of the one or more underlying portions of the environment 7104. In some embodiments, the starting point representation is a composition resulting from the graphical composition at time t ₃ in the scene 7038 in fig. 7G, optionally with greater color saturation, the appearance of the user interface element 7030-e being based on the resulting composition. In some embodiments, the starting point representation is a generated color gradient, optionally with greater color saturation, and the appearance of the user interface element 7040 at time t ₃ in fig. 7I is based on the generated color gradient. In fig. 7Y, the appearance of the background region 7086-a of the user interface element 7086 is generated by selectively applying a transform X _A, which is optionally a darkening transform (e.g., reducing color saturation), to a region represented by a starting point corresponding to a region included in the background region 7086-a. Another larger view of the effect of transformation X _A is shown in fig. 7Z, where panel 7090-a shows the result of transformation X _A applied to the entire starting point representation at time t ₃. In the example of FIG. 7Y, the transformation X _A is not applied to some other region of the user interface element 7086, such as regions 7086-b, 7086-c, and 7086-d.

For example, the appearance of the control region 7086-b is generated by selectively applying a transform X _B, which is optionally a brightening transform, to a region represented by a starting point corresponding to a region included in the control region 7086-b. Another larger view of the effect of transformation X _B is shown in fig. 7Z, where panel 7090-b shows the result of transformation X _B applied to the entire starting point representation of time t ₃. In the example of FIG. 7Y, the transform X _B is not applied to any of the regions 7086-a, 7086-c, and 7086-d.

Regions 7086-c and 7086-d of user interface element 7086 in FIG. 7Y illustrate two different transformation examples applied to text regions. The appearance of the text region 7086-c is the result of: the application of transform X _A is abandoned, the application of transform X _B is abandoned, and the application of transform X _D is abandoned (e.g., no transform is applied to the starting point representation, or a visual transform that is an identity transform is applied to the starting point representation). In other words, the appearance of the text region 7086-c is a result of maintaining the appearance of the start point transformation (e.g., maintaining the graphical composition or color gradient). Another larger view of the effect of giving up applying any transform to the start point representation (e.g., a larger view of the appearance of the start point representation) is shown in fig. 7Z, where panel 7090-c shows the start point representation at time t ₃ where no transform was applied. In some embodiments, the appearance of the text region 7086-c is the result of: the transform X _C described above with reference to fig. 7X is applied without applying any of transforms X _A, X _B, and X _D to text regions 7086-c, and optionally without applying transform X _C to any of regions 7086-a, 7086-b, and 7086-d.

In contrast, the appearance of the text region 7086-d is that the transformation X _D is selectively applied to the region represented by the starting point of time t ₃ that corresponds to the text character region included in the text region 7086-d (e.g., the transformation X _A is not applied and the transformation X _B is not applied). Another larger view of the effect of transformation X _D is shown in fig. 7Z, where panel 7090-d shows the result of transformation X _D applied to the entire starting point representation of time t ₃. In the example of FIG. 7Y, the transform X _D is not applied to any of the regions 7086-a, 7086-b, and 7086-c. Fig. 7X-7Y thus provide examples of how the appearance of a user interface element (e.g., user interface element 7030 or user interface element 7040) changes over time (e.g., from time t ₂ to time t ₃) by: the appearance of the user interface element is generated by applying visual transformations to different regions of the representation of the underlying content, where the representation itself is based on the graphical composition or color gradient of the underlying portion of the surrounding three-dimensional environment. One of ordinary skill in the art will recognize that because the representations change continuously over time, the appearance of the user interface element may be continuously generated by applying the same visual transformation to the representations of the underlying content (e.g., as described herein with reference to fig. 7A-7J).

In some embodiments, according to some embodiments, the input gestures used in the various examples and embodiments described herein (e.g., described with respect to fig. 7A-7Z and 8-14) optionally include discrete small motion gestures (which are performed by moving a user's hand finger relative to other finger or portion of the user's hand) without optionally requiring a larger mobile user's entire hand or arm to move away from its natural position and posture to perform operations immediately before or during making gestures for interacting with a virtual or mixed reality environment.

In some embodiments, the input gesture is detected by analyzing data or signals captured by a sensor system (e.g., sensor 190, FIG. 1; image sensor 314, FIG. 3). In some embodiments, the sensor system includes one or more imaging sensors (e.g., one or more cameras, such as a motion RGB camera, an infrared camera, and/or a depth camera). For example, the one or more imaging sensors are components of or provide data to a computer system (e.g., computer system 101 (e.g., portable electronic device 7100 or HMD) in fig. 1) that includes a display generation component (e.g., display generation component 120 (e.g., a touch screen display, stereoscopic display, or a display with a transparent portion) in fig. 1, 3, and 4) that functions as a display and a touch-sensitive surface. In some embodiments, the one or more imaging sensors include one or more rearward facing cameras on a side of the device opposite the display of the device. In some implementations, the input gesture is detected by a sensor system of the headset system (e.g., a VR headset that includes a stereoscopic display that provides a left image for a left eye of the user and a right image for a right eye of the user). For example, one or more cameras that are components of the head-mounted system are mounted on the front and/or underside of the head-mounted system. In some embodiments, one or more imaging sensors are located in a space in which the head-mounted system is used (e.g., arranged around the head-mounted system in various locations in a room) such that the imaging sensors capture images of the head-mounted system and/or a user of the head-mounted system. In some embodiments, the input gesture is detected by a sensor system of a head-up device (such as a head-up display, an automotive windshield having the ability to display graphics, a window having the ability to display graphics, a lens having the ability to display graphics). For example, one or more imaging sensors are attached to an interior surface of an automobile. In some embodiments, the sensor system includes one or more depth sensors (e.g., a sensor array). For example, the one or more depth sensors include one or more light-based (e.g., infrared) sensors and/or one or more sound-based (e.g., ultrasonic) sensors. In some embodiments, the sensor system includes one or more signal emitters, such as light emitters (e.g., infrared emitters) and/or sound emitters (e.g., ultrasonic emitters). for example, as light (e.g., light from an infrared light emitter array having a predetermined pattern) is projected onto a hand (e.g., hand 7102), an image of the hand under illumination of the light is captured by the one or more cameras and the captured image is analyzed to determine the position and/or configuration of the hand. Using signals from an image sensor pointing at the hand to determine an input gesture, rather than using signals from a touch-sensitive surface or other direct contact mechanism or proximity-based mechanism, allows the user to freely choose whether to perform a large motion or remain relatively stationary while providing an input gesture with his/her hand, without being subject to the limitations imposed by a particular input device or input area.

In some embodiments, the tap input is optionally a tap input of the thumb of the user's hand over the index finger (e.g., on a side of the index finger adjacent to the thumb). In some embodiments, tap input is detected without the need to lift the thumb from the side of the index finger. In some embodiments, the tap input is detected in accordance with determining a downward movement of the thumb followed by an upward movement of the thumb, wherein the thumb contacts the contact of the index finger for less than a threshold amount of time. In some implementations, the tap-hold input is detected in accordance with determining a time that the thumb is moved from the raised position to the stroked position and held in the stroked position for at least a first threshold amount of time (e.g., a tap time threshold or another time threshold that is longer than the tap time threshold). In some embodiments, the computer system requires that the hand as a whole remain substantially stationary in position for at least a first threshold amount of time in order to detect a tap-hold input by the thumb on the index finger. In some embodiments, touch-and-hold input is detected without requiring the hand as a whole to remain substantially stationary (e.g., the hand as a whole may move while the thumb rests on the side of the index finger). In some embodiments, a tap-hold-drag input is detected when the thumb touches the side of the index finger and the hand as a whole moves while the thumb rests on the side of the index finger.

In some implementations, the flick gesture is optionally a push or flick input by movement of the thumb across the index finger (e.g., from the palm side to the back side of the index finger). In some embodiments, the stretching movement of the thumb is accompanied by an upward movement of the side away from the index finger, for example, as in an upward flick input by the thumb. In some embodiments, during forward and upward movement of the thumb, the index finger moves in a direction opposite to that of the thumb. In some embodiments, the reverse flick input is performed by movement of the thumb from an extended position to a retracted position. In some embodiments, during the rearward and downward movements of the thumb, the index finger moves in a direction opposite to the direction of the thumb.

In some embodiments, the swipe gesture is optionally a swipe input by movement of the thumb along the index finger (e.g., along a side of the index finger adjacent to the thumb or on that side of the palm). In some embodiments, the index finger is optionally in an extended state (e.g., substantially straight) or a curled state. In some embodiments, during movement of the thumb in the swipe input gesture, the index finger moves between the extended state and the curled state.

In some embodiments, different phalanges (sometimes referred to as segments) of various fingers correspond to different inputs. Tap inputs of the thumb over various phalanges of various fingers (e.g., index finger, middle finger, ring finger, and optionally, pinky) are optionally mapped to different operations. Similarly, in some embodiments, different push or click inputs may be performed by a thumb across different fingers and/or different portions of the fingers to trigger different operations in the respective user interface contexts. Similarly, in some embodiments, different swipe inputs performed by the thumb along different fingers and/or in different directions (e.g., toward the distal or proximal ends of the fingers) trigger different operations in the respective user interface contexts.

In some implementations, the computer system treats tap input, flick input, and swipe input as different types of input based on the type of movement of the thumb. In some implementations, the computer system treats input having different finger positions tapped, touched, or swiped by a thumb as different sub-input types (e.g., proximal, middle, distal sub-types, or index, middle, ring, or little finger sub-types) of a given input type (e.g., tap input type, flick input type, or swipe input type). In some embodiments, the amount of movement performed by moving a finger (e.g., thumb) and/or other movement metrics associated with movement of the finger (e.g., speed, initial speed, ending speed, duration, direction, and/or movement pattern) are used to quantitatively affect the operation triggered by the finger input.

In some embodiments, the computer system identifies combination input types that combine a series of movements by the thumb, such as a tap-swipe input (e.g., a press of the thumb on the finger followed by a swipe along that side of the finger), a tap-flick input (e.g., a press of the thumb over the finger followed by a flick across the finger from the palm side to the back side of the finger), and/or a double-click input (e.g., two consecutive taps on that side of the finger at about the same location).

In some implementations, the gesture input is performed by the index finger instead of the thumb (e.g., the index finger performs a tap or swipe on the thumb, or the thumb and index finger move toward each other to perform a pinch gesture). In some implementations, wrist movement (e.g., flicking of the wrist in a horizontal or vertical direction) is performed immediately before the finger movement input, immediately after the finger movement input (e.g., for a threshold amount of time), or concurrently with the finger movement input, as compared to a finger movement input without a modification input by wrist movement, to trigger additional, different, or modified operations in the current user interface context. In some embodiments, a finger input gesture performed with a user's palm facing the user's face is considered a different type of gesture than a finger input gesture performed with a user's palm facing away from the user's face. For example, an operation performed with a flick gesture performed with a user palm facing the user's face has increased (or decreased) privacy protection compared to an operation (e.g., the same operation) performed in response to a flick gesture performed with a user palm facing away from the user's face.

Although in the examples provided in this disclosure one type of input or finger input may be used to trigger a certain type of operation, in other embodiments other types of input or finger input are optionally used to trigger the same type of operation.

Additional description regarding fig. 7A-7Z is provided below with reference to methods 8000, 9000, 10000, 11000, 12000, 13000, and 14000 described with respect to fig. 8-14 below.

FIG. 8 is a flowchart of a method 8000 of displaying a user interface element on a view of a physical environment, wherein the user interface element has an appearance based on a graphical composition of content in the physical environment behind the user interface element at a plurality of different points in time, according to some embodiments.

In some embodiments, method 8000 is performed at a computer system (e.g., computer system 101 in fig. 1) that is in communication with a display generating component (e.g., display generating component 120 in fig. 1,3, and 4) (e.g., heads-up display, touch screen, and/or projector) and optionally one or more input devices such as one or more cameras (e.g., a camera directed downward toward a user's hand (e.g., color sensor, infrared sensor, and/or other depth sensing camera) or directed forward from the user's head). In some embodiments, method 8000 is managed by instructions stored in a non-transitory computer readable storage medium and executed by one or more processors of a computer system (such as one or more processors 202 of computer system 101) (e.g., control unit 110 in fig. 1A). Some operations in method 8000 are optionally combined, and/or the order of some operations is optionally changed.

In some embodiments, method 8000 is performed at a computer system (e.g., computer system 101 in fig. 1, computer system 101 described with respect to fig. 7A-7G, 7H-7J, 7K-7O, 7P, 7Q-7W, and/or 7X-7Z). In some implementations, the computer system communicates with a display generating component (e.g., the first display generating component is a heads-up display, a head-mounted display (HMD), a display, a touch screen, a projector, etc.) and optionally one or more input devices (e.g., a camera, a controller, a touch-sensitive surface, a joystick, buttons, a glove, a watch, a motion sensor, and/or an orientation sensor). In some embodiments, the display generating component is the display generating component 7100 described with respect to fig. 7A-7Z. In some embodiments, the computer system is an integrated device having one or more processors and memory and at least some of the one or more input devices enclosed in the same housing as the display generating component. In some embodiments, the computer system includes a computing component (e.g., a server, a mobile electronic device such as a smart phone or tablet device, a wearable device such as a watch, wristband, earpiece, desktop computer, or laptop computer) that includes one or more processors and memory separate from the display generating component and/or one or more input devices. In some embodiments, the display generating component and the one or more input devices are integrated and packaged in the same housing. According to some embodiments, many of the features of method 8000 are described with respect to fig. 7A-7G.

In method 8000, the computer system simultaneously displays (8002) via the display generating component: a view of the physical environment; and a computer-generated user interface element overlaid on at least a portion of the view of the physical environment. The appearance of the computer-generated user interface element is based (8004) on the appearance of one or more portions of the view of the physical environment covered by the computer-generated user interface element. As described herein with reference to fig. 7A-7G, the display generation component 7100 displays an environment 7104 (fig. 7B) that includes a view of the physical environment 7102 (fig. 7A) and a computer-generated user interface element 7030 (fig. 7D) displayed as overlaid on at least a portion of the displayed view of the physical environment 7102 (e.g., overlaid on at least a portion of the environment 7104).

In method 8000, when a view of the physical environment and the computer-generated user interface element are displayed simultaneously, an appearance of a portion of the physical environment changes (8006). As described herein with reference to fig. 7B-7E, the box 7012 in the physical environment 7102 falls into space toward the floor 7008. In addition, in the transition from fig. 7B to fig. 7C, the viewing angle of the physical environment 7102 is shifted downward and backward; in the transition from fig. 7C to 7D, the viewing perspective is rotated (e.g., the computer system is rotated clockwise); and in the transition from fig. 7D to fig. 7E, the rotation of the viewing perspective is reversed (e.g., the computer system is rotated counterclockwise). In some embodiments, the computer system detects a change in appearance of the portion of the physical environment.

In method 8000, responsive to the change in appearance of the portion of the physical environment, the computer system updates (8008) an appearance of the computer-generated user interface element at a first time based on a graphical composition of the appearance of the one or more portions of the physical environment at different times prior to the first time, comprising: appearance of a first portion of the physical environment at a second time prior to the first time; and an appearance of a second portion of the physical environment at a third time prior to the second time. As described herein with reference to fig. 7D, in response to a change in the physical environment 7102 including the box 7012 falling within the physical environment 7102 and a change in the viewing perspective of the physical environment 7102, the computer system generates an appearance of the user interface element 7030 at time t ₂ based on the graphical composition of the appearances of the environment 7104 at different times t ₀ and t ₁ prior to time t ₂ (e.g., based on the appearance of the physical environment 7102). As described herein with reference to fig. 7E, in response to a change in the physical environment 7102 including the box 7012 further falling into the physical environment 7102 and a change in the viewing perspective of the physical environment 7102, the computer system updates the appearance of the user interface element 7030 at time t ₃ based on the graphical composition of the appearance of the environment 7104 at different times t ₁ and t ₂ prior to time t ₃ (e.g., based on the appearance of the physical environment 7102). In some embodiments, the second portion of the physical environment is the same as the first portion of the physical environment. In some implementations, the second portion of the physical environment is the same as the first portion of the physical environment (e.g., if the computer system moves relative to the physical environment between the first time and the second time).

In the case where the computer-generated user interface element is displayed on a view of one or more portions of the physical environment, basing the appearance of the computer-generated user interface element on the graphical composition of the appearance of the one or more portions of the physical environment, and updating the appearance of the computer-generated user interface element in response to changes in the one or more underlying portions of the physical environment simulates the appearance of objects made of partially transparent material (e.g., sometimes referred to as obscuring material) and helps the user understand what is in the physical environment and is obscured by the computer-generated user interface element, and informs the user of the changes occurring in the physical environment, which provides improved feedback to the user. In addition, the use of fuzzy material improves the user's situational awareness, which improves user safety by helping the user avoid collisions with physical objects in physical space, and reduces the likelihood that the user will suffer motion sickness when using the computer system.

In some embodiments, the appearance of the portion of the physical environment changes in response to movement of the computer system relative to the physical environment (e.g., changing the view of the physical environment from the perspective of the computer system, or the perspective of a camera or other imaging component or subsystem of the computer system). As described herein with reference to fig. 7B-7E, the appearance of the physical environment 7102 as displayed in the environment 7104 changes in response to movement (e.g., downward and rearward, via clockwise rotation, and/or via counterclockwise rotation) of the computer system relative to the physical environment 7102. Updating the appearance of computer-generated user interface elements in response to changes perceived or detected in the one or more underlying portions of the physical environment in response to movement of the computer system relative to the physical environment (e.g., in contrast to changes in the physical environment only when the computer system remains stationary) improves responsiveness of the system to changes in the physical environment and enables a user to interact with the computer system in more ways, which provides improved feedback to the user and provides additional control options without cluttering the user interface with additional display controls.

In some embodiments, the appearance of the portion of the physical environment changes in response to movement of one or more objects in the physical environment. In some embodiments, the one or more objects are different from the computer system. As described herein with reference to fig. 7B-7E, the appearance of the physical environment 7102 as displayed in the environment 7104 changes in response to movement of the bin 7012 in the physical environment 7102 (e.g., a bin falling into the physical environment), wherein the bin 7012 is different from the computer system. Updating the appearance of computer-generated user interface elements in response to movement of one or more objects in the physical environment improves the responsiveness of the system to changes occurring in the physical environment and informs the user of such changes, particularly where the changes occur in portions of the physical environment that are obscured from view by the computer-generated user interface elements, which provides improved feedback to the user.

In some embodiments, the computer-generated user interface element includes content (e.g., visual content such as text or graphics, and/or one or more selectable objects or controls), and the updating of the appearance of the computer-generated user interface element based on the graphical composition (e.g., the graphical composition of the appearance of the one or more portions of the physical environment) is applied to one or more regions (e.g., the background) of the computer-generated user interface element that are different from the content of the computer-generated user interface element. As described herein with reference to fig. 7D-7G, the user interface element 7030 includes a plurality of portions including a background portion 7030-a and content portions 7030-b and 7030-c, wherein the appearance of the background portion 7030-a is continuously updated based on and based on a graphical composition (e.g., also referred to as a composition), and wherein the appearance of the content portions 7030-b and 7030-c is optionally not based on the graphical composition. In some implementations, the appearance of the content of the computer-generated user interface element (e.g., the content portions 7030-b and 7030-c of the user interface element 7030 in fig. 7D) is not based on the appearance of one or more portions of the view of the physical environment. In some implementations, the appearance of the content of the computer-generated user interface element (e.g., the content portions 7030-b and 7030-c of the user interface element 7030 in fig. 7D) is not updated based on the graphical composition. Having the appearance of the background of the computer-generated user interface element based on the appearance of the underlying portion of the physical environment, rather than doing so with respect to the content of the computer-generated user interface element (such as visual content and/or selectable controls), improves the legibility of the content of the computer-generated user interface element while informing the user of changes occurring in the physical environment, which provides improved feedback to the user.

In some implementations, the graphic composition is based on a first amount (e.g., a first magnitude) of appearance of a first portion of the physical environment at a second time before the first time, and based on a second amount (e.g., a second magnitude) of appearance of a second portion of the physical environment at a third time before the second time; and the first amount is different (e.g., greater or less) than the second amount. As described herein with reference to scene 7036 in figure 7G, The appearance of the user interface element 7030-D at time t ₂ (which is the same as the appearance of the user interface element 7030 at time t ₂ in figure 7D) is based on a time t ₀ of the first portion of the physical environment 7102 that is prior to time t ₂ (e.g., Frame t ₀) and is based on a second portion of the physical environment 7102 at a time t ₁ (e.g., frame t ₁) prior to time t ₂, optionally in different amounts. Similarly, as described herein with reference to scene 7038 in FIG. 7G, The appearance of the user interface element 7030-E at time t ₃ (which is the same as the appearance of the user interface element 7030 at time t ₃ in figure 7E) is based on a time t ₁ of the first portion of the physical environment 7102 that is prior to time t ₃ (e.g., Frame t ₁) and is based on a second portion of the physical environment 7102 at a time t ₂ (e.g., frame t ₂) prior to time t ₃, optionally in different amounts. The graphical composition of the appearance of the computer-generated user interface element based on the appearance of the underlying portion of the physical environment (where the graphical composition is more based on the appearance of the physical environment at some times than at other times) informs the user of the changes that occur in the physical environment, where the appearance of the physical environment is emphasized more at times that are more relevant than at other times, which provides improved feedback to the user.

In some embodiments, at the first time, the contribution of the one or more portions of the physical environment to the appearance of the computer-generated user interface element at the first time is greater than the contribution of the one or more portions of the physical environment to the appearance of the computer-generated user interface element at the first time at a time that is further from the first time. As described herein with reference to scene 7036 in figure 7G, the contribution of the frame from time t ₂ to the appearance of the user interface element 7030-d at time t ₂ is greater than the contribution of the frame from time t ₁, which is different from time t ₂. In the same way as described above, The contribution of the frame from time t ₁ to the appearance of the user interface element 7030-d at time t ₂ is greater than the contribution of the frame from time t ₀ to the appearance of the user interface element 7030-d at time t ₂, Because time t ₀ is farther from time t ₂ than time t ₁ is from time t ₂. Similarly, as described herein with reference to scene 7038 in FIG. 7G, The contribution of the frame from time t ₃ to the appearance of the user interface element 7030-e at time t ₃ is greater than the contribution of the frame from time t ₂, which is different from time t ₃. In the same way as described above, The contribution of the frame from time t ₂ to the appearance of the user interface element 7030-e at time t ₃ is greater than the contribution of the frame from time t ₁ to the appearance of the user interface element 7030-e at time t ₃, Because time t ₁ is farther from time t ₃ than time t ₂ is from time t ₃. In some embodiments, the time closer to the first time is the time closest to the first time at which the one or more portions of the physical environment were sampled (e.g., the frame taken at time t ₃ is naturally the frame taken at the time closest in time to t ₃ and thus has the greatest contribution to the composition generated at time t ₃). Having the appearance of the computer-generated user interface element more based on the appearance of the physical environment at a time closer to the current point in time than the appearance of the physical environment at a time further from the current point in time better informs the user of the more recent changes occurring in the physical environment, which provides improved feedback to the user.

In some embodiments, the contribution of the one or more portions of the physical environment to the appearance of the computer-generated user interface at the first time at a time furthest from the first time is less than the contribution of the one or more portions of the physical environment to the appearance of the computer-generated user interface at the first time at any time closer to the first time. As described herein with reference to scene 7036 in fig. 7G, among frames t ₀、t₁ and t ₂, frame t ₀ contributes minimally to the appearance of user interface element 7030-d at time t ₂, Because t ₀ is the time furthest from t ₂ (e.g., among times t ₀、t₁ and t ₂). Similarly, as described herein with reference to scene 7038 in FIG. 7G, among frames t ₁、t₂ and t ₃, frame t ₁ contributes minimally to the appearance of user interface element 7030-e at time t ₃, Because t ₁ is the time furthest from t ₃ (e.g., among times t ₁、t₂ and t ₃). Minimizing the appearance of the computer-generated user interface element based on the appearance of the physical environment at a time furthest from the current point in time, as compared to the appearance of the physical environment at a time closer to the current point in time, informs the user of the changes occurring in the physical environment while de-emphasizing outdated information about the appearance of the physical environment, which provides improved feedback to the user.

In some embodiments, the graphical composition is based on the appearance of the one or more portions of the physical environment at a plurality of different times prior to the first time, and the plurality of different times is based on an amount of movement of the computer system relative to the physical environment (e.g., changing a view of the physical environment from a perspective of the computer system, or a perspective of a camera or other imaging component or subsystem of the computer system). As described herein with reference to fig. 7G, different numbers of frames may be included in generating a composition defining the appearance of the user interface element 7030 based on the amount of movement of the user's viewpoint between fig. 7B-7E. In some embodiments, the appearance of the computer-generated user interface element is updated based on the graphical composition of the appearance of the one or more portions of the physical environment at a first number of different times prior to the first time in accordance with a first amount of movement of the computer system relative to the physical environment; and updating the appearance of the computer-generated user interface element based on the graphical composition of the appearance of the one or more portions of the physical environment at a second number of different times prior to the first time in accordance with a second amount of movement of the computer system relative to the physical environment. In some embodiments, the number of different times is increased for a greater amount of movement of the computer system relative to the physical environment. In some embodiments, the number of different times is reduced for a greater amount of movement of the computer system relative to the physical environment. Basing the appearance of computer-generated user interface elements on the graphical composition of the appearance of underlying portions of the physical environment at different amounts of time for different amounts of movement of the computer system relative to the physical environment allows flexibility in balancing the computational burden of generating graphical compositions with the fidelity of changes occurring in the physical environment, which is associated with providing improved feedback to the user. For example, sampling the underlying physical environment more frequently for larger amounts of movement of the computer system may increase fidelity, which provides improved feedback to the user, but may also increase computational burden. In another example, sampling the underlying physical environment less frequently for larger amounts of movement may reduce the computational burden, but may affect fidelity.

In some embodiments, determining the graphic composition includes applying a noise pattern to the captured information (e.g., image data) about the appearance of the one or more portions of the physical environment at different times (e.g., as described above with reference to fig. 7G, specifically referring to user interface elements 7030-d in scene 7036 and user interface elements 7030-e in scene 7038). In some embodiments, applying the noise pattern includes deterministically changing the brightness and/or saturation of different portions of the image of the physical environment. In some embodiments, a sequence of pseudo-random noise patterns defining changes in brightness and/or saturation of different portions of an image is used. For example, a speckle-gray noise pattern (e.g., 10 frames, 1 frame every 1/60 second) resulting from an animation process between a plurality of different patterns over time is generated and optionally applied to the underlying image data using an image processing filter that increases image darkness if the noise pattern exceeds a predetermined threshold (e.g., 50% gray) and increases image brightness if the noise pattern is less than the predetermined threshold. Applying the noise pattern to the image data used to generate the graphic composition of the appearance of the physical environment at different points in time smoothes the data so as to reduce the impact of artifacts on the graphic composition and avoid producing too vivid graphic compositions (which would be visually discordant and would reduce the legibility of the content of the user interface element), thereby providing improved feedback to the user.

In some embodiments, in determining the graphical composition of the one or more portions of the physical environment at different times, the appearance of the first portion of the physical environment at the second time is offset (e.g., laterally offset or spatially offset) from the appearance of the second portion of the physical environment at the third time based on a difference between the perspective of the computer system at the second time and the perspective of the computer system at the third time. As described herein with reference to scene 7036 in fig. 7G, in generating a composition for the appearance of user interface elements 7030-d, the content from the different frames t ₀、t₁ and t ₂ is based on the user's content corresponding to frame t ₀ in fig. 7B, the different viewpoints corresponding to frame t ₁ in fig. 7C and corresponding to frame t ₂ in fig. 7D are offset from each other. Similarly, as described herein with reference to scene 7038 in fig. 7G, in generating a composition for the appearance of user interface element 7030-e, the content from the different frames t ₁、t₂ and t ₃ is based on the user's content corresponding to frame t ₁ in fig. 7C, The different viewpoints corresponding to frame t ₂ in fig. 7D and corresponding to frame t ₃ in fig. 7E are offset from each other. when the user view of the computer-generated user interface element changes relative to the underlying physical environment, and when the computer-generated user interface element changes orientation relative to the underlying physical environment, different portions of the physical environment enter the background of the computer-generated user interface element at different times. Generating a graphical composition of the appearance of the underlying portion of the physical environment by shifting those appearances based on different viewing perspectives of the underlying portion of the physical environment, such that a more realistic graphical composition of content is generated that is visible in the physical environment behind the computer-generated user interface element at different times, which helps the user better understand what is in the physical environment and what is obscured by the computer-generated user interface element, provides improved feedback to the user.

In some embodiments, determining the graphical composition of the appearance of the one or more portions of the physical environment at different times includes blurring a view of the physical environment captured by one or more cameras of the computer system (e.g., the appearance of a first portion of the physical environment at a second time is a blurred version of an image of the physical environment captured by the one or more cameras at a second time and/or the appearance of a second portion of the physical environment at a third time is a blurred version of an image of the physical environment captured by the one or more cameras at a third time). As described above with reference to fig. 7G, the composition upon which the appearance of the user interface elements 7030-d in the scene 7036 is generated optionally includes obscuring the appearance of frames t ₀、t₁ and/or t ₂, wherein frames t ₀、t₁ and/or t ₂ include portions of the view of the physical environment 7102 as captured by one or more cameras of the computer system. Similarly, the composition upon which the appearance of the user interface element 7030-e in the scene 7038 in fig. 7G is generated optionally includes obscuring the appearance of frames t ₁、t₂ and/or t ₃, wherein frames t ₁、t₂ and/or t ₃ include portions of the view of the physical environment 7102 as captured by one or more cameras of the computer system. Having the appearance of a computer-generated user interface element based on a blurred view of the physical environment as captured by one or more cameras of the computer system helps the user understand what is in the physical environment and what is obscured by the computer-generated user interface element, and informs the user of changes occurring in the physical environment, which provides improved feedback to the user.

In some embodiments, the graphic composition is partially translucent (e.g., the appearance of a first portion of the physical environment at a second time and/or the appearance of a second portion of the physical environment at a third time) as described herein, for example, with reference to fig. 7G. Having the appearance of the computer-generated user interface element based on the partially translucent graphical composition of the appearance of the physical environment at different times helps the user understand what is in the physical environment and what is obscured by the computer-generated user interface element and informs the user of the changes that occur in the physical environment, which provides improved feedback to the user.

In some embodiments, determining the graphical composition of the appearance of the one or more portions of the physical environment at different times includes reducing the resolution of information (e.g., image data) captured about the one or more portions of the physical environment at different times prior to the first time (e.g., as described above with reference to fig. 7G, specifically user interface elements 7030-d in scene 7036 and user interface elements 7030-e in scene 7038). In some embodiments, the appearance of the first portion of the physical environment at the second time is determined by reducing the resolution of information captured about the first portion of the physical environment at the second time. In some embodiments, the appearance of the second portion of the physical environment at the third time is determined by reducing the resolution of the information captured about the second portion of the physical environment at the third time. Reducing the resolution of information about the physical environment captured at different times in generating a graphical composition for the appearance of a computer-generated user interface element informs a user of changes occurring in the physical environment while reducing the computational burden of rendering the computer-generated user interface element, which provides improved feedback to the user.

In some embodiments, the appearance of the computer-generated user interface element is updated as the viewing perspective of the computer-generated user interface element (e.g., the user's point of view) changes (e.g., movement of the computer system changes the view of the physical environment covered by the computer-generated user interface element from the perspective of one or more cameras of the computer system), or movement of the user (which changes the user's perspective of the computer-generated user interface element and in some cases also changes the user's perspective of the physical environment).

In some implementations, the appearance of the computer-generated user interface element is updated as the computer-generated user interface element moves (e.g., relative to a displayed view of the physical environment). As described herein with reference to fig. 7E, the appearance of the user interface element 7030 changes in response to the user's point of view rotating relative to the environment 7104 and the user interface element 7030 remaining in the same position relative to the user's point of view (thereby causing the user interface element 7030 to correspondingly rotate relative to the environment 7104 and the physical environment 7102). Updating the appearance of the computer-generated user interface element as the computer-generated user interface element moves relative to the displayed view of the physical environment helps the user understand what is in the physical environment and what is obscured by the computer-generated user interface element, which provides improved feedback to the user.

In some embodiments, the appearance of the computer-generated user interface element is updated based on the one or more additional computer-generated user interface elements having a simulated location that is behind the simulated location of the computer-generated user interface element (e.g., from the perspective of the user, in the context of the computer-generated user interface element). As described herein with reference to fig. 7B-7G, the appearance of user interface element 7030 (e.g., in addition to the appearance based on the representation of a physical object such as box 7012) is updated based on the appearance of virtual ball 7108 and virtual wall hanging 7106 in environment 7104, optionally while user interface element 7030 has a simulated position in front of the simulated positions of virtual ball 7108 and virtual wall hanging 7106, such as simulated position 7030-1 or simulated position 7030-2 in fig. 7F. Updating the appearance of the computer-generated user interface element based on other computer-generated user interface elements positioned in a virtual sense behind the computer-generated user interface element helps the user understand what is still in the environment that the user is viewing and what is obscured by the computer-generated user interface element (e.g., a three-dimensional environment in which various computer-generated user interface elements are virtually positioned and overlaid on a view of the physical environment), which provides improved feedback to the user.

In some embodiments, the appearance of the one or more additional computer-generated user interface elements is updated over time (e.g., similar to that described herein with reference to the computer-generated user interface elements). As described herein with reference to fig. 7B-7G, the appearance of virtual ball 7108 and virtual wall-mount 7106 also updates over time according to changes in environment 7104 (e.g., movement of virtual ball 7108 and/or changes in viewing perspective of environment 7104). In some embodiments, the appearance of other virtual objects in environment 7104 (such as virtual ball 7108, virtual wall hanging 7106, and/or one or more additional user interface elements similar to user interface element 7030) is also updated over time similar to user interface element 7030 based on the composition of the underlying portion of environment 7104. Updating the appearance of multiple computer-generated user interface elements over time based on those and other computer-generated user interface element changes and changes in physical environment gives the computer-generated user interface element a consistent appearance that simulates the appearance of objects made of partially transparent material, which helps the user understand the environment that the user is looking at and the spatial relationship between different objects in the environment, which provides improved feedback to the user.

In some embodiments, the computer-generated user interface element is a back surface of the displayed user interface that includes one or more selectable user interface objects (and optionally content). For example, as described herein with reference to fig. 7D, the user interface element 7030 may be a user interface of an application program or a system user interface displayed within the environment 7104. Updating the appearance of the back surface of the displayed user interface overlaid on the view of the physical environment (where the updating is based on changes in the physical environment) helps the user understand what is in the physical environment and what is obscured by the computer-generated user interface element, which provides improved feedback to the user.

In some embodiments, the appearance of the computer-generated user interface element is based on the appearance of one or more portions of the view of the physical environment covered by the computer-generated user interface element, regardless of the simulated position of the computer-generated user interface element relative to the physical environment. As described herein with reference to fig. 7F-7G, the appearance of the user interface element 7030-d in fig. 7G is based on the appearance of the overlapping portion of the view of the physical environment 7102 (e.g., as part of environment 7104) regardless of whether the simulated position of the user interface element 7030-d is position 7030-1, position 7030-2, position 7030-3, or position 7030-4 in fig. 7F. Similarly, the appearance of the user interface element 7030-e in FIG. 7G is based on the appearance of the overlapping portion of the view of the physical environment 7102 (e.g., as part of the environment 7104) regardless of whether the simulated position of the user interface element 7030-e is position 7030-1, position 7030-2, position 7030-3, or position 7030-4 in FIG. 7F. In some embodiments, the appearance of the computer-generated user interface element is based on the appearance of a portion of the physical environment that is in front of and/or behind the simulated location of the computer-generated user interface element. Having the appearance of a computer-generated user interface element based on the appearance of overlapping portions of the physical environment (including portions that may be in front of the simulated position of the computer-generated user interface element from the perspective of the user) helps the user understand what is in the physical environment in the vicinity of the computer-generated user interface element, which provides improved feedback to the user.

In some embodiments, the simulated position of the computer-generated user interface element is a first distance from a viewpoint of the user (e.g., the simulated position of the viewpoint), and the appearance of the computer-generated user interface element is based on an appearance of one or more portions of the physical environment that are a second distance from the viewpoint of the user (e.g., the simulated position of the viewpoint), wherein the first distance is greater than the second distance (e.g., the appearance of the computer-generated user interface element is based on an appearance of one or more portions of the physical environment that are closer to the viewpoint of the user than the simulated position of the computer-generated user interface element). As described herein with reference to fig. 7F-7G, even when the simulated position of the user interface element 7030-d is position 7030-4, the appearance of the user interface element 7030-d may still be as shown in scene 7036 in fig. 7G and based on the appearance of the overlapping portion of the environment 7104 that is more proximate to the viewpoint 7032 of the user than the position 7030-4 is proximate to the viewpoint 7032. Similarly, even when the simulated position of the user interface element 7030-e is position 7030-4, the appearance of the user interface element 7030-e may still be as shown in scene 7038 in FIG. 7G and based on the appearance of an overlapping portion of the environment 7104 that is more proximate to the viewpoint 7032 of the user than the position 7030-4 is proximate to the viewpoint 7032. Having the appearance of a computer-generated user interface element based on the appearance of overlapping portions of the physical environment (including portions that may be in front of the simulated position of the computer-generated user interface element from the perspective of the user) helps the user understand what is in the physical environment in the vicinity of the computer-generated user interface element, which provides improved feedback to the user.

It should be understood that the particular order in which the operations in fig. 8 are described is merely an example and is not intended to suggest that the order is the only order in which the operations may be performed. Those of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein with respect to other methods described herein (e.g., methods 9000, 10000, 11000, 12000, 13000, and 14000) apply in a similar manner to method 8000 described above with respect to fig. 8. For example, the user interface elements or objects (including their appearance and regions), representations of physical objects and/or physical objects, virtual objects, graphical compositions/compositions, movements, three-dimensional environments (including their views and appearances), display generating components, and/or points in time described above with reference to method 8000 optionally have one or more of the characteristics of the user interface elements or objects (including their appearance and regions), representations of physical objects and/or physical objects, virtual objects, graphical compositions/compositions, movements, three-dimensional environments (including their views and appearances), display generating components, and/or points in time described herein with reference to other methods described herein (e.g., methods 9000, 10000, 11000, 12000, 13000, and 14000). For the sake of brevity, these details are not repeated here.

FIG. 9 is a flowchart of a method 9000 of displaying a user interface element on a view of a three-dimensional environment, where the user interface element has an appearance of a color gradient generated based on one or more colors sampled from a plurality of points in the three-dimensional environment, according to some embodiments.

In some embodiments, method 9000 is performed at a computer system (e.g., computer system 101 in fig. 1) in communication with a display generation component (e.g., display generation component 120 in fig. 1,3, and 4) (e.g., heads-up display, touch screen, and/or projector) and optionally one or more input devices such as one or more cameras (e.g., a camera directed downward toward a user's hand (e.g., color sensor, infrared sensor, and/or other depth sensing camera) or directed forward from the user's head). In some embodiments, method 9000 is managed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors of a computer system, such as one or more processors 202 of computer system 101 (e.g., control unit 110 in fig. 1A). Some of the operations in method 9000 are optionally combined and/or the order of some of the operations is optionally changed.

In some embodiments, method 9000 is performed at a computer system (e.g., computer system 101 in fig. 1, computer system 101 described with respect to fig. 7A-7G, 7H-7J, 7K-7O, 7P, 7Q-7W, and/or 7X-7Z). In some implementations, the computer system communicates with a display generating component (e.g., the first display generating component is a heads-up display, a head-mounted display (HMD), a display, a touch screen, a projector, etc.) and optionally one or more input devices (e.g., a camera, a controller, a touch-sensitive surface, a joystick, buttons, a glove, a watch, a motion sensor, and/or an orientation sensor). In some embodiments, the display generating component is the display generating component 7100 described with respect to fig. 7A-7Z. In some embodiments, the computer system is an integrated device having one or more processors and memory and at least some of the one or more input devices enclosed in the same housing as the display generating component. In some embodiments, the computer system includes a computing component (e.g., a server, a mobile electronic device such as a smart phone or tablet device, a wearable device such as a watch, wristband, earpiece, desktop computer, or laptop computer) that includes one or more processors and memory separate from the display generating component and/or one or more input devices. In some embodiments, the display generating component and the one or more input devices are integrated and packaged in the same housing. According to some embodiments, many of the features of method 9000 are described with respect to fig. 7H to 7J.

In method 9000, the computer system concurrently displays (9002) via the display generating component: a view of the physical environment; and a computer-generated user interface element overlaid on at least a portion of the view of the physical environment. The appearance of the computer-generated user interface element is based (9004) on a respective gradient between a first set of representative colors sampled from one or more portions of the view of the physical environment covered by the computer-generated user interface element. The first set of representative colors includes (9006): a first color representing a portion of a view of the physical environment (e.g., selected or sampled from) behind a first portion of the computer-generated user interface element (e.g., its simulated location) within a first period of time (e.g., at a first time, or during a first time range); and a second color (e.g., different from the first color) representing a portion of the view of the physical environment (e.g., selected or sampled from) behind a second portion of the computer-generated user interface element (e.g., its simulated location) within a first period of time (e.g., at the same first time as the time for the first color, or during the same first time range as the time range for the first color). For example, as described herein with reference to fig. 7H-7J, the appearance of the user interface element 7040 is based on a color gradient between representative colors sampled from the region 7042 (fig. 7H), the region 7046 (fig. 7I), and the region 7048 (fig. 7J) of the environment 7104. Representative colors include colors 7044-1 through 7044-3 (FIG. 7H), colors 7044-4 through 7044-6 (FIG. 7I), and colors 7044-7 through 7044-10 (FIG. 7J).

In method 9000, an appearance of a portion of the physical environment changes when a view of the physical environment and the computer-generated user interface element are displayed simultaneously (9008). For example, as described herein with reference to fig. 7H-7J, the view of environment 7104 changes in response to changes in the user's point of view (e.g., tilting and leveling the computer system in fig. 7H-7I) and movement of user interface element 7040 in environment 7104 (fig. 7J). In some embodiments, the computer system detects a change in appearance of the portion of the physical environment.

In method 9000, in response to a change in appearance of the portion of the physical environment (e.g., in addition to updating the view of the physical environment accordingly), the computer system updates (9010) the appearance of the computer-generated user interface element based on a respective gradient between a second set of representative colors sampled from one or more portions of the view of the physical environment covered by the computer-generated user interface element. The second set of representative colors includes (9012): a third color (e.g., different from the first color) representing a portion of the view of the physical environment (e.g., selected or sampled from) behind the first portion of the computer-generated user interface element (e.g., its simulated location) for a second period of time (e.g., at a second time, or during a second time range) different from (after) the first period of time; and a fourth color (e.g., different from the second color and/or the third color) representing a portion of the view of the physical environment that is behind (e.g., is selected or sampled from) a second portion of the computer-generated user interface element (e.g., its simulated location) within a second time period (e.g., at a second time that is the same as the time for the third color, or during a second time range that is the same as the time range for the third color). For example, in response to a change in the appearance of environment 7104 from fig. 7H to fig. 7I, the appearance of user interface element 7040 is updated to be based on the gradient between colors 7044-4 to 7044-6 in fig. 7I, rather than based on the gradient between colors 7044-1 to 7044-3 as in fig. 7H. Similarly, in response to a change in the appearance of environment 7104 from fig. 7I to 7J, the appearance of user interface element 7040 is updated (e.g., further) to be based on the gradient between colors 7044-7 to 7044-10 in fig. 7J, rather than based on the gradient between colors 7044-4 to 7044-6 as in fig. 7I. In some embodiments, the appearance of the computer-generated user interface element is maintained (e.g., continuing to be based on the respective gradients between the first set of representative colors, as described herein with reference to fig. 7H-7I) in the absence of a change in the appearance of a portion of the physical environment and any other computer-generated user interface element that overlaps the computer-generated user interface element.

In the case where a computer-generated user interface element is displayed on a view of one or more portions of a physical environment, basing the appearance of the computer-generated user interface element on gradients between colors sampled from the one or more underlying portions of the physical environment, and updating the appearance of the computer-generated user interface element in response to changes in the one or more underlying portions of the physical environment, simulates the appearance of an object made of partially transparent material (e.g., sometimes referred to as obscuring material) and helps a user understand what is in the physical environment and what is obscured by the computer-generated user interface element, and informs the user of the changes occurring in the physical environment, while reducing the computational burden of rendering the computer-generated user interface element relative to the more complex graphical composition of the appearance of the underlying portions of the physical environment, which provides improved feedback to the user. In addition, the use of fuzzy material improves the user's situational awareness, which improves user safety by helping the user avoid collisions with physical objects in physical space, and reduces the likelihood that the user will suffer motion sickness when using the computer system.

In some embodiments, the computer-generated user interface element is opaque (e.g., as described herein with reference to fig. 7H-7J). Displaying the computer-generated user interface element as opaque and having an appearance based on a gradient between colors sampled from the one or more underlying portions of the physical environment helps a user understand what is in the physical environment and what is obscured by the computer-generated user interface element and informs the user of changes occurring in the physical environment while reducing the computational burden of rendering the computer-generated user interface element relative to rendering the computer-generated user interface element with a degree of transparency and/or a more complex graphical composition of the appearance of the underlying portions of the physical environment, which provides improved feedback to the user.

In some embodiments, the first set of representative colors includes a fifth color representing a portion of the view of the physical environment that is behind (e.g., is selected or sampled from) the third portion of the computer-generated user interface element within a first time period (e.g., within a first time range that is the same as the time for the first color and the second color) or within a first time range that is the same as the time range for the first color and the second color), and the second set of representative colors includes a sixth color representing a portion of the view of the physical environment that is behind (e.g., is within a second time period (e.g., within a second time range that is the same as the time range for the third color and the fourth color) the third portion of the computer-generated user interface element (e.g., Selected or sampled therefrom). In some implementations, the first set of representative colors includes three or more colors of portions of the view representing the physical environment that are behind respective corresponding portions of the computer-generated user interface element over a first period of time. In some implementations, the second set of representative colors includes three or more colors of portions of the view representing the physical environment that are behind respective corresponding portions of the computer-generated user interface element over a second period of time. For example, as described herein with reference to fig. 7H, the appearance of the user interface element 7040 at time t ₂ is based on a color gradient that includes at least three colors 7044-1 through 7044-3 that are sampled from behind at least three corners of the user interface element 7040, respectively. similarly, as described herein with reference to fig. 7I, the appearance of the user interface element 7040 at time t ₃ is based on a color gradient that includes at least three colors 7044-4 through 7044-6 that are sampled from behind at least three corners of the user interface element 7040, respectively. Likewise, as described herein with reference to FIG. 7J, the appearance of the user interface element 7040 at time t ₄ is based on a color gradient that includes at least three colors 7044-7, 7044-8, and/or 7044-10 and 7044-9, respectively, sampled from behind at least three corners of the user interface element 7040. Displaying the computer-generated user interface element as having an appearance that is updated in response to changes in the one or more underlying portions of the physical environment based on a gradient between at least three colors sampled from the one or more underlying portions of the physical environment helps the user better understand what is in the physical environment and what is obscured by the computer-generated user interface element, and better informs the user of changes occurring in the physical environment by giving the computer-generated user interface element an appearance that is more representative of the underlying physical environment (e.g., versus whether two colors or only one color is used), which provides improved feedback to the user.

In some implementations, the appearance of the computer-generated user interface element based on the respective gradients extends to one or more edges (e.g., all edges, or at least all displayed edges) of the computer-generated user interface element. For example, as described herein with reference to fig. 7H-7J, the appearance of the user interface element 7040 is based on a color gradient up to the edge of the user interface element 7040. Applying the respective gradients to the appearance of the computer-generated user interface element extending to the visible edge of the computer-generated user interface element simulates the appearance of an object made of uniform material, which allows the user to understand what is in the physical environment and what is obscured by the computer-generated user interface element, and informs the user of the changes occurring in the physical environment, while avoiding giving the computer-generated user interface element a visually discordant appearance in the context of the view of the physical environment, which provides improved feedback to the user.

In some embodiments, the colors of the respective portions of the computer-generated user interface elements are sampled from respective portions of the view of the physical environment that are behind the simulated locations of the respective portions of the computer-generated user interface elements (e.g., in a simulated three-dimensional space, such as an augmented or virtual reality environment that includes a view of the physical environment, where different computer-generated user interface elements have different respective simulated locations in the three-dimensional space). For example, as described herein with reference to fig. 7H-7J, the appearance of the user interface element 7040 is based on a color gradient between representative colors sampled from the region 7042 (fig. 7H), region 7046 (fig. 7I), and region 7048 (fig. 7J) of the environment 7104 that is behind the user interface element 7040. Specifically, for example, in fig. 7H, the color 7044-1 of the upper left corner of the user interface element 7040 is sampled from bin 7012'; color 7040-2 in the upper right hand corner of user interface element 7044 is sampled from wall hanging 7106; and the color 7040-3 of the bottom edge region of the user interface element 7044 is sampled from the wall 7004'. Similarly, for example, in FIG. 7I, the color 7044-4 in the upper left corner of the user interface element 7040 is sampled from bin 7012'; color 7040-5 in the upper right hand corner of user interface element 7044 is sampled from wall hanging 7106; and the color 7040-6 of the bottom edge region of the user interface element 7044 is sampled from the wall 7004'. Likewise, for example, in FIG. 7J, color 7044-7 in the upper left corner of user interface element 7040 is sampled from wall hanging 7106; the color 7044-8 in the upper right hand corner of the user interface element 7040 and the color 7044-10 in the lower left hand corner of the user interface element 7040 are sampled from the wall 7004'; and the color 7044-9 in the lower right hand corner of the user interface element 7040 is sampled from the bin 7014'. Displaying the computer-generated user interface element as having an appearance based on a gradient between colors sampled from the one or more underlying portions of the physical environment simulates the appearance of an object made of partially transparent material and helps the user understand what is in the physical environment and what is obscured by the computer-generated user interface element, and informs the user of changes occurring in the physical environment, which provides improved feedback to the user.

In some embodiments, when a viewing perspective of a computer-generated user interface element (e.g., a user's point of view) changes relative to a physical environment (e.g., movement of a computer system that changes a view of the physical environment from a perspective of one or more cameras of the computer system), or movement of a user that changes a perspective of the user to the computer-generated user interface element and/or the physical environment), an update to an appearance of the computer-generated user interface element based on the respective gradient changes the appearance of the computer-generated user interface element. For example, as described herein with reference to fig. 7H-7I, as the view of environment 7104 changes in response to changes in the viewpoint based on the user of the tilting computer system (fig. 7H) and leveling computer system (fig. 7I), the appearance of user interface element 7040 is updated. Displaying the computer-generated user interface element as having an appearance that is updated as the point of view of the user changes (e.g., the viewing perspective of the computer-generated user interface element and/or the one or more underlying portions of the physical environment changes) based on the gradient between colors sampled from the one or more underlying portions of the physical environment provides a more realistic and immersive viewing experience that helps the user understand what is in the physical environment and what is obscured by the computer-generated user interface element, which provides improved feedback to the user.

In some implementations, updating the appearance of the computer-generated user interface element based on the respective gradients includes changing the appearance of the computer-generated user interface element as the computer-generated user interface element moves (e.g., relative to a displayed view of the physical environment). For example, as described herein with reference to fig. 7J, as the view of the environment 7104 changes in response to movement of the user interface element 7040 in the environment 7104, the appearance of the user interface element 7040 is updated. Displaying the computer-generated user interface element as having an appearance that is updated as the computer-generated user interface element moves relative to the displayed view of the physical environment based on the gradient between colors sampled from the one or more underlying portions of the physical environment helps the user understand what is in the physical environment and what is obscured by the computer-generated user interface element, which provides improved feedback to the user.

In some embodiments, the appearance of the computer-generated user interface element is updated based on the one or more additional computer-generated user interface elements having a simulated location that is behind the simulated location of the computer-generated user interface element (e.g., from the perspective of the user, in the context of the computer-generated user interface element). For example, the first set of representative colors includes one or more colors representing portions of the one or more additional computer-generated user interface elements that are virtually positioned behind the computer-generated user interface element during the first period of time, and/or the second set of representative colors includes one or more colors representing portions of the one or more additional computer-generated user interface element that are virtually positioned behind the computer-generated user interface element during the second period of time. For example, as described herein with reference to fig. 7H-7J, the color gradient for the appearance of user interface element 7040 is based at least in part on one or more colors sampled from computer-generated wall-hanging 7106 behind user interface element 7040 in environment 7104. In some embodiments, the appearance of the computer-generated user interface element is updated based on one or more physical objects in the physical environment having respective physical locations in the physical environment that are behind the simulated location of the computer-generated user interface element (e.g., in addition to or instead of having a simulated location behind the simulated location of the computer-generated user interface element based on the one or more additional computer-generated user interface elements). Displaying the computer-generated user interface element as having an appearance based on a gradient between colors sampled from other computer-generated user interface elements positioned in a virtual sense behind the computer-generated user interface element helps the user understand what is still and is obscured by the computer-generated user interface element in the environment that the user is viewing (e.g., a three-dimensional environment in which various computer-generated user interface elements are virtually positioned and overlaid on a view of the physical environment), which provides improved feedback to the user.

In some embodiments, the appearance of the one or more additional computer-generated user interface elements is updated over time (e.g., similar to that described herein with reference to the computer-generated user interface elements). For example, as described herein with reference to fig. 7H-7J, in some embodiments, the appearance of wall hanging 7106 is updated over time in a similar manner to user interface element 7040. In another example, as described herein with reference to fig. 7K, the appearance of the user interface element 7052 is also updated over time in a similar manner to other user interface elements described herein with reference to fig. 7A-7J and/or the user interface element 7050 in fig. 7K, the user interface element 7052 being behind the user interface element 7050 and in some embodiments serving as a basis for the appearance of the user interface element 7050. In some embodiments, the appearance of the respective additional computer-generated user interface element is maintained (e.g., continuing to be based on the respective gradient between the third set of representative colors) without any change in the appearance of the portion of the physical environment and any other computer-generated user interface element that overlaps the respective additional computer-generated user interface element. Displaying the plurality of computer-generated user interface elements as having an appearance based on gradients between colors sampled from the underlying physical environment and any other underlying computer-generated user interface elements gives the computer-generated user interface elements a consistent appearance that simulates the appearance of objects made of partially transparent material, which helps the user understand the environment the user is viewing and the spatial relationship between different objects in the environment, which provides improved feedback to the user.

In some embodiments, the appearance of a respective additional computer-generated user interface element of the one or more additional computer-generated user interface elements is based on a respective gradient (e.g., the same gradient for the computer-generated user interface element) between a third set of representative colors sampled from one or more portions of the view of the physical environment covered by the respective additional computer-generated user interface element (e.g., sampled from one or more portions of the physical environment behind the simulated location of the respective additional computer-generated user interface element). In some embodiments, the third set of representative colors comprises: a seventh color representing a portion of the view of the physical environment (e.g., selected or sampled from) behind the first portion of the corresponding additional computer-generated user interface element (e.g., its simulated position) within a third time period (e.g., at a third time, or during a third time range); and an eighth color (e.g., different from the seventh color) representing a portion of the view of the physical environment (e.g., selected or sampled from) behind the second portion of the corresponding additional computer-generated user interface element (e.g., its simulated position) within a third time period (e.g., at a third time that is the same as the time for the seventh color, or during a third time range that is the same as the time range for the seventh color). In some embodiments, the third time period is a first time period, a second time period, or a time period different from the first time period and the second time period.

In some embodiments, when a view of the physical environment (e.g., and at least a portion of the computer-generated user interface element and/or the corresponding additional computer-generated user interface element) is displayed, the appearance of a portion of the physical environment changes. In some embodiments, the computer system detects a change in appearance of the portion of the physical environment. In some embodiments, in response to a change in appearance of the portion of the physical environment (e.g., in addition to updating the view of the physical environment accordingly), the computer system updates the appearance of the respective additional computer-generated user interface element based on a respective gradient between a fourth set of representative colors sampled from one or more portions of the view of the physical environment covered by the respective additional computer-generated user interface element. In some embodiments, the fourth set of representative colors comprises: a ninth color (e.g., different from the seventh color) representing a portion of the view of the physical environment (e.g., selected or sampled from) behind the first portion of the additional corresponding computer-generated user interface element (e.g., its simulated position) for a fourth period of time (e.g., at a fourth time, or during a fourth range of times) that is different from (after) the third period of time; and a tenth color (e.g., different from the eighth color and/or the ninth color) representing a portion of the view of the physical environment (e.g., selected or sampled from) behind a second portion of the corresponding additional computer-generated user interface element (e.g., its simulated location) during a fourth time period (e.g., at a fourth time that is the same as the time for the ninth color, or during a fourth time range that is the same as the time range for the ninth color). For example, as described herein with reference to fig. 7H-7J, in some embodiments, the appearance of wall hanging 7106 also updates over time in a similar manner to user interface element 7040. In another example, as described herein with reference to fig. 7K, in some embodiments, the appearance of the user interface element 7052 is also updated over time in a similar manner to the user interface element 7050. In some embodiments, the fourth time period is the second time period (e.g., if the third time period is the first time period) or a different time period than the first time period, the second time period, and the third time period. Displaying additional computer-generated user interface elements as having an appearance based on gradients between colors sampled from the underlying physical environment and any other underlying computer-generated user interface elements gives the computer-generated user interface elements a consistent appearance that simulates the appearance of objects made of partially transparent material, which helps the user understand the environment the user is looking in and the spatial relationship between different objects in the environment, which provides improved feedback to the user.

In some embodiments, the respective colors of the portions of the view representing the physical environment that are behind the respective portions of the computer-generated user interface element are sampled from a second computer-generated user interface element having a simulated position that is at least partially behind the simulated position of the respective portions of the computer-generated user interface element. For example, in some cases, the first color is selected or sampled from a portion of the second computer-generated user interface element that is behind the first portion of the computer-generated user interface element (e.g., rather than selecting or sampling a color for each computer-generated user interface element directly from a view of the physical environment). In some cases, the second color is selected or sampled from a portion of the third (e.g., second) computer-generated user interface element that is behind the second portion of the computer-generated user interface element. Similarly, in some cases, the third color and/or the fourth color is selected or sampled from portions of other computer-generated user interface elements that are behind the aforementioned computer-generated user interface elements.

For example, as described herein with reference to fig. 7H-7J, if the appearance of wall hanging 7106 is updated over time in a similar manner as described for user interface element 7040 in fig. 7H-7J, the appearance of the portion of user interface element 7040 that overlaps wall hanging 7106 is optionally based on or re-uses the generated appearance of the underlying portion of wall hanging 7106, rather than being separately generated from the portions of the environment that are behind both wall hanging 7106 and user interface element 7040. Similarly, as described herein with reference to fig. 7K, the appearance of the foreground user interface element 7050 is optionally based on or re-uses the generated appearance of the underlying portion of the background user interface element 7052. Having the appearance of the computer-generated user interface element based on the color gradient of the underlying computer-generated user interface element gives the computer-generated user interface element a consistent appearance that simulates the appearance of an object made of partially transparent material, which helps the user understand what is still in the environment that the user is looking at and what is obscured by the computer-generated user interface element in the foreground, while reducing the computational burden of generating the color gradient and/or obscuring the appearance of each computer-generated user interface element, respectively, which provides improved feedback to the user.

In some embodiments, the appearance of the computer-generated user interface element is based on a respective gradient between a respective set of representative colors sampled from the one or more portions of the view of the physical environment, irrespective of the simulated position of the computer-generated user interface element relative to the physical environment. For example, as described herein with reference to fig. 7H-7J, the appearance of the user interface element 7040 is optionally based on a portion of the environment 7104 that is in front of the simulated position (e.g., position 7030-4 in fig. 7F) of the user interface element 7040 in the environment 7104. In some embodiments, the respective gradients use colors sampled from portions of the physical environment that are in front of and/or behind the simulated location of the computer-generated user interface element. Having the appearance of the computer-generated user interface element based on a gradient between colors sampled from overlapping portions of the physical environment, including portions that may be in front of the simulated position of the computer-generated user interface element from the perspective of the user, helps the user understand what is in the physical environment in the vicinity of the computer-generated user interface element, which provides improved feedback to the user.

It should be understood that the particular order in which the operations in fig. 9 are described is merely an example and is not intended to suggest that the order is the only order in which the operations may be performed. Those of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein with respect to other methods described herein (e.g., methods 8000, 10000, 11000, 12000, 13000, and 14000) apply in a similar manner to method 9000 described above with respect to fig. 9. For example, the user interface elements or objects (including their appearance and regions), the physical objects and/or representations of physical objects, virtual objects, user viewpoints, movements, three-dimensional environments (including their views and appearances), display generating components, and/or color gradients described above with reference to method 9000 optionally have one or more of the characteristics of the user interface elements or objects (including their appearances and regions), the physical objects and/or representations of physical objects, virtual objects, user viewpoints, movements, three-dimensional environments (including their views and appearances), display generating components, and/or color gradients described herein with reference to other methods described herein (e.g., methods 8000, 10000, 11000, 12000, 13000, and 14000). For the sake of brevity, these details are not repeated here.

FIG. 10 is a flowchart of a method 10000 of displaying a user interface element having an appearance of simulated material with simulated thickness and curvature along an edge of the user interface element, where the edge of the user interface element has simulated optical properties that interact with light in a three-dimensional environment, according to some embodiments.

In some embodiments, method 10000 is performed at a computer system (e.g., computer system 101 in fig. 1) that communicates with a display generation component (e.g., display generation component 120 in fig. 1,3, and 4) (e.g., heads-up display, touch screen, and/or projector) and optionally one or more input devices such as one or more cameras (e.g., a camera directed downward toward a user's hand (e.g., color sensor, infrared sensor, and/or other depth sensing camera) or directed forward from the user's head). In some embodiments, method 10000 is managed by instructions stored in a non-transitory computer readable storage medium and executed by one or more processors of a computer system, such as one or more processors 202 of computer system 101 (e.g., control unit 110 in fig. 1A). Some operations in method 10000 are optionally combined and/or the order of some operations is optionally changed.

In some embodiments, method 10000 is performed at a computer system (e.g., computer system 101 in fig. 1, computer system 101 described with respect to fig. 7A-7G, 7H-7J, 7K-7O, 7P, 7Q-7W, and/or 7X-7Z). In some implementations, the computer system communicates with a display generating component (e.g., the first display generating component is a heads-up display, a head-mounted display (HMD), a display, a touch screen, a projector, etc.) and optionally one or more input devices (e.g., a camera, a controller, a touch-sensitive surface, a joystick, buttons, a glove, a watch, a motion sensor, and/or an orientation sensor). In some embodiments, the display generating component is the display generating component 7100 described with respect to fig. 7A-7Z. In some embodiments, the computer system is an integrated device having one or more processors and memory and at least some of the one or more input devices enclosed in the same housing as the display generating component. In some embodiments, the computer system includes a computing component (e.g., a server, a mobile electronic device such as a smart phone or tablet device, a wearable device such as a watch, wristband, or earpiece, a desktop computer, or a laptop computer) that includes one or more processors and memory separate from the display generating component and/or the one or more input devices. In some embodiments, the display generating component and the one or more input devices are integrated and packaged in the same housing. According to some embodiments, many of the features of method 10000 are described with respect to fig. 7K-7O.

In method 10000, the computer system displays (10002) a first view of the three-dimensional environment via a display generation component, including displaying a first user interface object in the first view of the three-dimensional environment and displaying background content behind the first user interface object in the three-dimensional environment. The first user interface object includes (10004) a first portion having an appearance that is selected based on an appearance of background content (e.g., a representation of virtual content or physical space) that is located behind the first portion of the first user interface object in the three dimensional environment. In some implementations, the first portion of the first user interface object is a back surface of the first user interface object. The first user interface object has a simulated thickness between the first portion and a front of the first user interface object; and the content is displayed between a first portion of the first user interface object and a front portion of the first user interface object. For example, as shown in 7K and described herein with reference to this figure, the user interface element 7050 has a simulated thickness indicated by edges 7050-e, wherein the content is displayed as having an appearance that is located behind a front surface of the user interface element 7050 and between the front and back surfaces of the user interface element 7050 (e.g., embedded within the user interface element 7050 at a depth that is less than the simulated thickness of the user interface element 7050); likewise, the user interface element 7052 optionally has a simulated thickness indicated by edges 7052-e, wherein the content is displayed having an appearance that is behind the front surface of the user interface element 7052 and embedded within the user interface element 7052. As shown in fig. 7L-7O and described herein with reference to these figures, the user interface element 7054 has a simulated thickness between a front surface of the user interface element 7054 and a back surface of the user interface element 7054, wherein the user interface element 7054 is configured to display content in a background portion 7054-a between the front surface and the back surface of the user interface element 7054.

When the first user interface object is displayed, the computer system detects (10006) a request to move content (e.g., scroll, reposition, resize, rescale, rotate, and/or otherwise transform). In response to detecting the request to move the content, the computer system moves (10008) the content within the first user interface object while the content remains between the first portion of the first user interface object and the front of the first user interface object. Moving the content within the first user interface object includes (10010) displaying a visual effect applied to the content based on the simulated thickness of the first user interface object. For example, as described herein with reference to fig. 7K, content embedded within the user interface element 7050 and within a threshold distance of the edge 7050-e is displayed as having an optical effect wrapped around the simulated thickness of the edge 7050-e according to simulated internal reflection and/or refraction. Similarly, as described herein with reference to fig. 7K, content embedded within the user interface element 7052 and within a threshold distance of the edge 7052-e is displayed as having an optical effect wrapped around the simulated thickness of the edge 7052-e according to simulated internal reflection and/or refraction.

Having the appearance of a user interface object in a three-dimensional environment based on the appearance of background content behind the user interface object in the three-dimensional environment (e.g., in a virtual sense) simulates the appearance of an object made of transparent or partially transparent material (e.g., sometimes referred to as a blur material), which appearance is enhanced by: content moving within the user interface object is displayed with a visual effect applied to the edge of the user interface object to simulate content moving within the object having simulated thickness and refractive properties along the edge of the object (e.g., simulate internal reflections from the edge of the object, such as internal reflections of light and/or embedded content), which helps the user understand what is still in the three-dimensional environment and what is still obscured by the user interface object, which provides improved feedback to the user. In addition, the use of fuzzy material improves the user's situational awareness, which improves user safety by helping the user avoid collisions with physical objects in physical space, and reduces the likelihood that the user will suffer motion sickness when using the computer system.

In some embodiments, the appearance of the first user interface object (e.g., a first portion thereof) is based on the appearance of the background content according to a first level of translucency of the first user interface object (e.g., so as to simulate the appearance of the background content to be visible through translucent material having the first level of translucency), and the method includes displaying a shadow on the background content displayed adjacent to an edge of the first user interface object, wherein the appearance of the shadow is according to a second level of translucency of the first user interface object (e.g., so as to simulate the shadow cast on the background content by translucent material having the second level of translucency), the second level of translucency being different from the first level of translucency. For example, while the user interface element 7050 in fig. 7K is shown as having the appearance of an object made of a transparent or partially transparent material, what is shown in environment 7104 is that the shadow cast by the user interface element 7050 is optionally different from the shadow to be cast from a physical object made of the same transparent or partially transparent material. For example, when a physical pane of transparent glass may cast a given amount of shadow on nearby objects in the physical environment, user interface element 7050 is optionally displayed to simulate a pane of transparent glass while also being displayed to have the appearance of casting more or less shadow on nearby objects in environment 7104 than the physical pane of glass casting on those nearby objects. Similarly, in some embodiments, user interface element 7052 in fig. 7K is optionally displayed to cast more or less shadow in environment 7104 than a corresponding physical object of the same material that user interface element 7052 is displayed to simulate.

In the physical world, a physical object casts an associated amount of shadows on other objects, the associated amount of shadows being defined by the illumination in the physical environment and the materials comprising the object (e.g., including materials having a particular degree of transparency or translucence). In a computer-generated three-dimensional environment, a computer-generated object may be displayed with a transparent or semi-transparent amount that is independent of the amount of shadow that the computer-generated object is shown to cast on other objects. Displaying a user interface object that simulates an object made of transparent or partially transparent material while displaying the object as an independent amount of shadows cast on other objects provides the user with information about the size and location of the user interface object in a three-dimensional environment, as well as providing flexibility in how such information is conveyed in order to help orient the user in the three-dimensional environment, which provides improved feedback to the user.

In some implementations, displaying the visual effect applied to the content based on the simulated thickness of the first user interface object includes: portions of the content that are within a threshold distance from an edge of the first user interface object are displayed as simulated thickness bends around the first user interface object (e.g., as described herein with reference to user interface elements 7050 and 7052 of fig. 7K). The display of content moving within the user interface object as having a visual effect applied to the edges of the user interface object to simulate content bending around the simulated thickness of the user interface object enhances the appearance of the user interface object made of transparent or partially transparent material with refractive properties by simulating internal reflection from the edges of the object and transparency of the edges of the object, which helps the user understand what is in a three-dimensional environment and what is obscured by the user interface object, which provides improved feedback to the user.

In some embodiments, one or more edges of the first user interface object are displayed with curvature (e.g., curved or rounded edges) and with a visual effect of reflecting light from the one or more edges. For example, as described herein with reference to fig. 7K, the user interface element 7050 is optionally displayed with a visual effect that reflects light from one or more locations along the curved edge 7050-e of the user interface element 7050 and/or the user interface element 7052 is displayed with a visual effect that reflects light from one or more locations along the curved edge 7050-e of the user interface element 7052. In another example, as described herein with reference to fig. 7L-7O, the user interface element 7054 has a simulated thickness and is displayed with a visual effect of reflecting light from one or more locations along an edge of the user interface element 7054, which edge is optionally curved. In some implementations, a portion of the first user interface object that is not an edge is displayed without curvature and/or without a visual effect of reflected light. Displaying the visual effect of reflecting light from the edges of the user interface object enhances the appearance of a user interface object made of transparent or partially transparent material with refractive properties by simulating internal reflections from the edges of the object, which helps the user understand what is in a three-dimensional environment and what is obscured by the user interface object, which provides improved feedback to the user.

In some embodiments, the curvature is in two dimensions (e.g., curvature in a plane, such as in the x-direction and y-direction or in any other two-dimensional coordinate system). For example, as described herein with reference to fig. 7K, the edges of the user interface element 7050 and the edges of the user interface element 7052 are rounded in at least two dimensions. In another example, as described herein with reference to fig. 7L-7O, the edges of the user interface element 7054 are optionally rounded in two dimensions (e.g., the edges where two faces (such as top and side, side and bottom, front and side, or side and back) meet are rounded such that the two faces smoothly transition into each other). Displaying the visual effect of reflecting light in two dimensions from the edges of a user interface object enhances the appearance of a user interface object made of transparent or partially transparent material having refractive properties by simulating internal reflections from the edges of the object and helps the user distinguish the user interface object from a representation or rendering of a three-dimensional physical object in a three-dimensional environment, which provides improved feedback to the user.

In some embodiments, the curvature is in three dimensions (e.g., curvature in space, such as curvature in the x-direction, y-direction, and z-direction, or any other three-dimensional coordinate system). For example, as described herein with reference to fig. 7K, the edges of the user interface element 7050 and/or the edges of the user interface element 7052 are optionally rounded in three dimensions. In another example, as described herein with reference to fig. 7L-7O, edges and/or corners of the user interface element 7054 are optionally rounded in three dimensions (e.g., vertices at which three faces intersect are rounded such that three faces smoothly transition into each other). Displaying the visual effect of reflecting light in three dimensions from the edges of a user interface object enhances the appearance of a user interface object made of transparent or partially transparent material having refractive properties by simulating internal reflections from the edges of the object and helps the user visualize the user interface object as a realistic three-dimensional object in a three-dimensional environment, which provides improved feedback to the user.

In some embodiments, the three-dimensional environment includes a representation of at least a portion of a physical environment (e.g., in which one or more cameras of the computer system are located), and the visual effect of reflecting light from the one or more edges of the first user interface object is based on light detected in the physical environment. For example, as described herein with reference to fig. 7K, specular reflection of user interface elements 7050 and/or 7052 in fig. 7K may simulate reflection of physical light from one or more physical light sources in physical environment 7102 (fig. 7A). In another example, as described herein with reference to fig. 7L-7O, specular reflection from user interface element 7054 is based, in some embodiments, at least in part, on illumination in physical environment 7102, which is represented, at least in part, by environment 7104. Displaying a visual effect of light reflected from edges of the user interface object based on actual light detected from a physical environment represented by the displayed three-dimensional environment helps the user visualize the user interface object as a realistic three-dimensional object in the physical environment and reduces cognitive imbalance between the user interface object and a view of the physical environment, which provides improved feedback to the user.

In some embodiments, the visual effect of reflecting light from the one or more edges of the first user interface object is based on light in a three-dimensional environment. For example, as described herein with reference to fig. 7K, specular reflection of user interface elements 7050 and/or 7052 in fig. 7K may simulate reflection of light from one or more simulated light sources in environment 7104 (e.g., different from and not corresponding to actual light in physical environment 7102). In another example, as described herein with reference to fig. 7L-7O, specular reflection from the user interface element 7054 is, in some embodiments, based at least in part on simulated light in the environment 7104 (e.g., different from and not corresponding to actual light in the physical environment 7102). In some embodiments, the light in the three-dimensional environment includes a representation of simulated light from a virtual light source in the three-dimensional environment and/or light from a physical environment represented by the three-dimensional environment. Displaying a visual effect of light reflected from an edge of the user interface object based on the simulated light in the three-dimensional environment helps the user visualize the user interface object as a three-dimensional object in the three-dimensional environment and increases the immersion in the three-dimensional environment, which provides improved feedback to the user.

In some embodiments, the first user interface object includes one or more activatable user interface objects, and the one or more activatable user interface objects are displayed on a front portion of the first user interface object (e.g., on a surface of the front portion of the first user interface object, rather than between the front portion and the first portion within the first user interface object). For example, as described herein with reference to fig. 7L-7O, the user interface object 7054 includes raised areas 7054-d which appear as portions of material that are raised relative to the front surface of the user interface element 7054, similar to a separate piece of material that is placed on top of the front surface of the user interface element 7054. In some embodiments, the one or more activatable user interface objects are displayed on a surface (e.g., a plane) that is a predetermined distance from a front of the first user interface object (e.g., above or in front of the front surface of the first user interface object). Displaying the activatable user interface object on the front surface of the user interface object distinguishes the activatable object from content (e.g., a quantity of material such as multimedia content and/or text content) that is displayed in a manner embedded within the user interface object and provides visual feedback to the user that the activatable object is interactive, which provides improved feedback to the user.

In some implementations, the first activatable user interface object of the first user interface object has a first thickness. In some embodiments, a computer system detects an input interacting with a first activatable user interface object; and in response to detecting the input, displaying the first activatable user interface object at a second thickness different from the first thickness. For example, as described herein with reference to fig. 7M-7N, the thickness of the raised region 7054-d changes in response to user interaction with the raised region 7054-d (e.g., the raised region 7054-d is displayed at a first thickness in fig. 7M, while the raised region 7054-d is displayed at a second thickness different from that in fig. 7M in response to the user 7002 focusing on the raised region 7054-d). For example, in the absence of interaction with the first activatable user interface object, the first activatable user interface object is displayed extending a first amount from a corresponding surface (such as a front portion of the first user interface object). In this example, in response to interaction with the first activatable user interface object, the first activatable user interface object is displayed as extending a different amount from the respective surface and optionally extending in a different direction (e.g., from protruding (e.g., extending forward or upward) from the first user interface object to recessed (e.g., extending backward or downward) into the first user interface object). Changing the thickness of the activatable user interface object relative to the surface of the underlying user interface object provides visual feedback to the user: having detected an interaction with the activatable object and being or to be responsive to further interactions to perform an operation associated with the activatable object provides improved feedback to the user.

In some implementations, the computer system displays a shadow corresponding to the first user interface object on the background content displayed adjacent to the first user interface object. In some implementations, a computer system detects input that interacts with a first user interface object (e.g., to move the first user interface object in a simulated three-dimensional space); and in response to detecting an input interacting with the first user interface object, changing an appearance of a shadow corresponding to the first user interface object (e.g., to indicate a change in thickness, distance, or height of the first user interface object relative to the background). For example, as described herein with reference to fig. 7M-7N, longer shadows from the raised regions 7054-d are displayed on the user interface element 7054 in fig. 7N in response to the user 7002 focusing on the raised regions 7054-d (e.g., changing in response to the user gazing at the raised regions 7054-d according to the thickness of the raised regions 7054-d). In some implementations, the appearance of the shadow indicates a simulated distance between a first user interface object (e.g., a first point in or on the first user interface object in simulated three-dimensional space) and background content (e.g., a second point in or on the background content in simulated three-dimensional space), and the shadow is displayed as having a first appearance when the first user interface object is a first distance from the background content, and is displayed as having a second appearance different from the first appearance when the first user interface object is a second distance from the background content (e.g., in response to user input moving the first user interface object in simulated three-dimensional space and relative to the background content). Changing the appearance of the shadow of the activatable user interface object, particularly where the appearance change is performed in conjunction with changing the thickness of the activatable user interface object relative to the surface of the underlying user interface object, enhances the activatable user interface object as a three-dimensional appearance and provides visual feedback to the user that interaction with the activatable object has been detected, which provides improved feedback to the user.

In some implementations, the first view of the three-dimensional environment includes a view of a front of the first user interface object. In some implementations, a computer system receives a request to display a view of a back of a first user interface object (e.g., a request to rotate or flip the first user interface object within a three-dimensional environment, or a request to view the three-dimensional environment from a different perspective with a view of the back of the first user interface object, e.g., due to movement of a user and/or the one or more cameras of the computer system relative to a physical environment represented by the three-dimensional environment, the request corresponding to changing a viewpoint of the user from being in front of and facing the first user interface object to being behind and facing the first user interface object). In some implementations, in response to a request to display a view of the back of the first user interface object, the computer system visually weakens the content between the first portion of the first user interface object and the front of the first user interface object (e.g., by blurring, fading, shading, or darkening, or other similar transformations, or by dropping or ceasing the display). For example, as described herein with reference to fig. 7K, content displayed as embedded within user interface element 7050 and/or user interface element 7052 is optionally visible from a front of user interface elements 7050 and 7052, such as from the perspective shown in fig. 7K, and is not clearly visible from a back of user interface elements 7050 and 7052. In some embodiments, the front portion of the first user interface object has the appearance of a transparent material through which content within the first user interface object is visible. In some embodiments, the back of the first user interface object has the appearance of a translucent material through which the content is less visible than from the front of the first user interface object. In some implementations, while viewing the back of the first user interface object, the content is not visible (e.g., not shown), even though the appearance of the first user interface object as viewed from the back continues to be based on the appearance of the background content in the three-dimensional environment behind the first user interface object (e.g., portions of the three-dimensional environment behind the first user interface object are at least partially visible through the first user interface object). Weakening the content embedded in the user interface object when the user interface object is viewed from the back avoids the undesirable effects of displaying the content in a manner that is opposite to the manner in which the content appears in an actual transparent or semi-transparent object, which reduces the computational burden of rendering the user interface object and provides improved feedback to the user.

In some embodiments, the computer system displays the first user interface object as having a first appearance according to a first context of the three-dimensional environment (e.g., a first time of day or other environmental conditions in the three-dimensional environment); and in response to a change of the three-dimensional environment to a second context different from the first context, displaying the first user interface object as having a second appearance different from the first appearance according to the second context of the three-dimensional environment (e.g., the first time of day or other environmental conditions in the three-dimensional environment). For example, as described herein with reference to fig. 7K-7M, the specular reflection, brightness, color temperature, and other visual properties of the user interface elements 7050, 7052, and/or 7054 are optionally adjusted to correspond to the context of the environment 7104. Changing the appearance of the user interface object based on the context of the surrounding three-dimensional environment provides visual feedback to the user regarding the change in the surrounding environment (e.g., changes in color, illumination, and/or time of day), which helps the user understand the environment the user is viewing, which provides improved feedback to the user.

In some implementations, the computer system selects an appearance of the first user interface object based on a context of the three-dimensional environment. In some implementations, in accordance with determining that the three-dimensional environment is in the first context, selecting a first appearance based on the first context of the three-dimensional environment; and in accordance with a determination that the three-dimensional environment is in a second context different from the first context, selecting a second appearance based on the second context of the three-dimensional environment. The second appearance is different from the first appearance. For example, as described herein with reference to fig. 7K-7M, the specular reflection, brightness, color temperature, and other visual properties of the user interface elements 7050, 7052, and/or 7054 are optionally selected based on the context of the environment 7104. For example, the color of the respective portion of the first user interface object is selected based on or in accordance with one or more colors in the three-dimensional environment (e.g., one or more colors of content or objects positioned behind the simulated position of the first user interface object in the three-dimensional environment and/or the ambient color temperature of the three-dimensional environment). If the warmer color is more prevalent in the three-dimensional environment, the first user interface object has a warmer-toned appearance. If the colder color is more prevalent in the three-dimensional environment, the first user interface object has a colder-toned appearance. In another example, the brightness of the first user interface object is selected based on or in accordance with the brightness and/or amount of light in the three-dimensional environment (e.g., based on simulated illumination in the three-dimensional environment and/or light in a physical environment represented by the three-dimensional environment). The first user interface object has a brighter appearance if the three-dimensional environment is bright. If the three-dimensional environment is dim, the first user interface object has a darker appearance. Selecting the appearance of the user interface object based on the context of the surrounding three-dimensional environment reduces cognitive imbalance in which the user interface object appears in an improper location and helps the user understand the environment that the user is viewing, which provides improved feedback to the user.

It should be understood that the particular order in which the operations in fig. 10 are described is merely an example and is not intended to suggest that the order is the only order in which the operations may be performed. Those of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein with respect to other methods described herein (e.g., methods 8000, 9000, 11000, 12000, 13000, and 14000) apply in a similar manner to method 10000 described above with respect to fig. 10. For example, the user interface elements or objects (including their appearance and regions), the physical objects and/or representations of physical objects, virtual objects, user viewpoints, movements, three-dimensional environments (including their views and appearances), display generating components, light sources, reflections, and/or shadows described above with reference to method 10000 optionally have one or more of the characteristics of the user interface elements or objects (including their appearances and regions), the physical objects and/or representations of physical objects, virtual objects, user viewpoints, movements, three-dimensional environments (including their views and appearances), display generating components, light sources, reflections, and/or shadows described herein with reference to other methods described herein (e.g., methods 8000, 9000, 11000, 12000, 13000, and 14000). For the sake of brevity, these details are not repeated here.

FIG. 11 is a flow chart of a method 11000 of displaying a user interface element having an appearance of simulated material with simulated thickness and curvature along an edge of the user interface element, where the edge of the user interface element has simulated optical properties that interact with light in a three-dimensional environment, according to some embodiments.

In some embodiments, method 11000 is performed at a computer system (e.g., computer system 101 in fig. 1) that includes or communicates with a display generating component (e.g., display generating component 120 in fig. 1,3, and 4) (e.g., heads-up display, touch screen, and/or projector) and one or more input devices such as one or more cameras (e.g., a camera directed downward toward a user's hand (e.g., color sensor, infrared sensor, and/or other depth sensing camera) or directed forward from the user's head). In some embodiments, method 11000 is managed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors of a computer system, such as one or more processors 202 of computer system 101 (e.g., control unit 110 in fig. 1A). Some operations in method 11000 are optionally combined and/or the order of some operations is optionally changed.

In some embodiments, method 11000 is performed at a computer system (e.g., computer system 101 in fig. 1, computer system 101 described with respect to fig. 7A-7G, 7H-7J, 7K-7O, 7P, 7Q-7W, and/or 7X-7Z). In some implementations, the computer system communicates with a display generating component (e.g., the first display generating component is a heads-up display, a head-mounted display (HMD), a display, a touch screen, a projector, etc.) and optionally one or more input devices (e.g., a camera, a controller, a touch-sensitive surface, a joystick, buttons, a glove, a watch, a motion sensor, and/or an orientation sensor). In some embodiments, the display generating component is the display generating component 7100 described with respect to fig. 7A-7Z. In some embodiments, the computer system is an integrated device having one or more processors and memory and at least some of the one or more input devices enclosed in the same housing as the display generating component. In some embodiments, the computer system includes a computing component (e.g., a server, a mobile electronic device such as a smart phone or tablet device, a wearable device such as a watch, wristband, or earpiece, a desktop computer, or a laptop computer) that includes one or more processors and memory separate from the display generating component and/or the one or more input devices. In some embodiments, the display generating component and the one or more input devices are integrated and packaged in the same housing. According to some embodiments, many of the features of method 11000 are described with respect to fig. 7K-7O.

In method 11000, a computer system displays (11002) a user interface via a display generation component, the user interface comprising a view of a three-dimensional environment (e.g., a view corresponding to a portion of a physical environment in a field of view of one or more cameras of the computer system); and simulated three-dimensional objects in a three-dimensional environment (e.g., simulated objects that are not representations or renderings of physical objects in the field of view of one or more cameras of a computer system at the same time), the simulated three-dimensional objects being displayed with appearances corresponding to respective simulated materials. The first region of the simulated three-dimensional object is displayed as having an appearance that a portion of the respective simulated material is removed from the surface of the simulated three-dimensional object (e.g., a recessed portion or pit in the respective simulated material of the simulated three-dimensional object, such as a portion etched or cut away from the surface of the simulated three-dimensional object). For example, as described herein with reference to fig. 7L-7O, the user interface element 7054 includes a recessed region 7054-b and a recessed region 7054-c. In some embodiments, a region of the simulated three-dimensional object outside (e.g., adjacent) the first region has a first simulated thickness, and a first portion of the simulated three-dimensional object has a second simulated thickness that is less than the first simulated thickness (e.g., material is removed from the first portion). For example, the recessed region 7054-b and the recessed region 7054-c are thinner than the background region 7054-a of the user interface element 7054 in fig. 7L-7N. In some embodiments, imparting an appearance that a portion of the respective simulated material has been removed from the surface of the simulated three-dimensional object is achieved by: the stereoscopic depth of the display region and/or the specular effect such as reflection and/or refraction from one or more edges of the region (e.g., displaying a greater amount of light reflection from the edges of the etched deeper regions). In some implementations, the apparent depth (e.g., apparent etch depth) of the region from which material has been removed is small compared to the size of the simulated three-dimensional object. For example, while the depth of the region may correspond to 10% -90% removal of the thickness of the simulated three-dimensional object, the depth of the region is a fraction (e.g., less than 1% -20%) of the thickness, length, and/or width of the simulated three-dimensional object. The computer system also displays (11002), via the display generation component, a first user interface element comprising a first region of the simulated three-dimensional object (e.g., the first region of the simulated three-dimensional object identifies the first user interface element; the first region of the simulated three-dimensional object is the first user interface element). For example, as described herein with reference to fig. 7L-7O, the recessed region 7054-b, recessed region 7054-c, and raised region 7054-d of the user interface element 7054 correspond to interactive user interface elements.

In method 11000, a computer system detects (11004) a first input directed to a first user interface element. In response to detecting a first input (11006) directed to the first user interface element, the computer system performs (11008) a respective operation associated with the first user interface element in the user interface. For example, as described herein with reference to fig. 7L-7O, the computer system is configured to perform respective operations associated with respective ones of the recessed region 7054-b, the recessed region 7054-c, or the raised region 7054-d in response to user input directed to the respective regions. Displaying one or more regions of the simulated three-dimensional object having the appearance of simulated material as portions of the simulated material removed from the surface of the simulated object and associating those regions with user interface elements, distinguishing those regions from other regions along the surface of the simulated object, and providing visual feedback to the user that can interact with those regions to perform operations in the user interface, provides improved feedback to the user.

In some embodiments, one or more edges of the simulated three-dimensional object (e.g., edges along the outline of the simulated three-dimensional object, and/or edges of the etched area such as the first user interface element) are displayed with a visual effect of light from the three-dimensional environment (e.g., from one or more light sources) being reflected from the one or more edges based on the properties of the respective simulated material. For example, as described herein with reference to fig. 7L-7O, specular reflection is displayed along one or more edges of the user interface element 7054. In some embodiments, the visual effect of the reflected light is based on light from one or more (e.g., physical and/or virtual) light sources. In some embodiments, the visual effect is based on light from multiple (e.g., physical and/or virtual) light sources. Displaying one or more edges of the simulated three-dimensional object as having specular reflection (including specular reflection from both the outer and inner surfaces of the edge of the simulated object) simulates the appearance of a physical object made of transparent or partially transparent material (e.g., sometimes referred to as a blur material), which provides the user with visual feedback regarding the location of the light source in the three-dimensional environment and distinguishes other displayed representations of the simulated object from objects corresponding to the physical object in the user's surrounding physical environment, which provides the user with improved feedback and improves the user's situational awareness, which improves user safety by helping the user avoid collisions with the physical object in physical space, and reduces the likelihood that the user will suffer from motion sickness when using the computer system.

In some embodiments, the light from the three-dimensional environment includes simulated light from one or more simulated light sources in the three-dimensional environment. For example, as described herein with reference to fig. 7L-7O, specular reflection based on the simulated light source in environment 7104 is displayed along one or more edges of user interface element 7054. Displaying one or more edges of the simulated three-dimensional object as having specular reflection based on one or more simulated light sources in the three-dimensional environment allows for the generation and adjustment of lighting effects in the environment by the computer system, which provides control over the context and atmosphere of the user's viewing experience, which provides improved feedback to the user.

In some embodiments, the three-dimensional environment corresponds to a physical environment (e.g., in which one or more cameras of a computer system are located), and the light from the three-dimensional environment includes light detected in the physical environment (e.g., from one or more physical light sources such as the sun, a lamp or luminaire, a flashlight, a headlight or headlamp, a flame or fire, or other physical light source). For example, as described herein with reference to fig. 7L-7O, specular reflection based on physical light and/or physical light sources in the physical environment 7102 (fig. 7A) is displayed along one or more edges of the user interface element 7054. In some embodiments, a representation of at least a portion of the physical environment is displayed in a three-dimensional environment. Displaying one or more edges of the simulated three-dimensional object as having specular reflection based on one or more physical light sources in a physical environment corresponding to the three-dimensional environment provides visual feedback to the user regarding illumination in the user's surrounding physical environment, which provides improved feedback to the user and improves the user's situational awareness, which increases user safety by helping the user avoid collisions with physical objects in the physical space, and reduces the likelihood that the user will suffer from motion sickness when using the computer system.

In some embodiments, the simulated three-dimensional object is displayed at a first location in the three-dimensional environment, and the one or more edges of the simulated three-dimensional object have an appearance based on a visual effect of light reflected from the one or more edges according to the simulated three-dimensional object being in the first location. In some embodiments, a computer system detects an input to move a simulated three-dimensional object in a three-dimensional environment. In some embodiments, in response to detecting an input to move a simulated three-dimensional object in a three-dimensional environment, the computer system: displaying the simulated three-dimensional object at a second location in the three-dimensional environment different from the first location; and changing the appearance of the one or more edges of the simulated three-dimensional object based on the visual effect of light reflected from the one or more edges according to the simulated three-dimensional object being in the second position.

For example, on a simulated transparent or partially transparent object positioned below a light source (e.g., a simulated or physical light source) from the viewpoint of a user, specular reflection is displayed along a top edge of the simulated object (e.g., representing light reflected outward from the top edge of the simulated object) and, to a lesser extent, specular reflection is displayed along a bottom edge of the simulated object (e.g., representing light that has traveled through the simulated object and reflected inward from the bottom edge of the simulated object). If the simulated object is moved to a position to the right of the light source from the viewpoint of the user, specular reflection is instead displayed along the left edge of the simulated object (e.g., representing light reflected outward from the left edge of the simulated object) and, to a lesser extent, along the right edge of the simulated object (e.g., representing light that has traveled through the simulated object and reflected inward from the right edge of the simulated object). In some implementations, as the simulated object moves gradually from a position below the light source to a position to the right of the light source, specular reflection on the top edge of the simulated object moves gradually to the left along the top edge of the simulated object and then downward along the left edge of the simulated object, and specular reflection on the bottom edge of the simulated object moves gradually to the right along the bottom edge of the simulated object and then upward along the right edge of the simulated object, depending on where light will reflect outward and inward from the edge of the simulated object. In another example, as described herein with reference to fig. 7L-7M, the appearance and position of the specular reflection from the user interface element 7054 changes as the user interface element 7054 moves from the position in fig. 7L to the position shown in fig. 7M.

Displaying a change in specular reflection from the one or more edges of the simulated three-dimensional object in response to movement of the simulated object in the three-dimensional environment gives the simulated object an appearance of how the simulated object interacts with the physical object and provides visual feedback to the user regarding the position of the light source in the three-dimensional environment and the spatial relationship between the simulated object, the light source and the viewpoint of the user, which improves the situational awareness of the user and reduces cognitive imbalance between the simulated object and the display environment, which provides improved feedback to the user.

In some embodiments, the view of the three-dimensional environment is from a first viewpoint of the user relative to the three-dimensional environment, and the one or more edges of the simulated three-dimensional object have an appearance based on visual effects of light reflected from the one or more edges according to the first viewpoint of the user (e.g., the appearance of light reflected from the one or more edges is as it would appear to a user viewing the simulated three-dimensional object in the three-dimensional environment from the viewpoint of the first viewpoint). In some embodiments, transitioning the computer system to a second viewpoint of the user relative to the three-dimensional environment (e.g., in response to detecting movement of the viewpoint of the user, such as movement via the computer system or one or more cameras of the computer system, or in response to detecting input requesting a change in the viewpoint of the user, such as an input to zoom or pan a view of the three-dimensional environment), includes: updating the user interface to a view of the three-dimensional environment from the second viewpoint of the user (e.g., gradually transitioning the view of the three-dimensional environment to simulate a continuous movement from the first viewpoint to the second viewpoint); and changing an appearance of the one or more edges of the simulated three-dimensional object based on a visual effect of light reflected from the one or more edges according to a second viewpoint of the user (e.g., the appearance of light reflected from the one or more edges as it appears to a user viewing the simulated three-dimensional object in the three-dimensional environment from the viewpoint of the second viewpoint).

For example, as described herein with reference to fig. 7L-7M, the appearance and location of specular reflection from the user interface element 7054 will change as the viewpoint of the user with respect to the user interface element 7054 changes. Displaying a change in specular reflection from the one or more edges of the simulated three-dimensional object in response to a change in the viewpoint of the user relative to the simulated object and/or the three-dimensional environment gives the simulated object an appearance of how the physical object interacts with the light from different viewing perspectives of the physical object, and provides visual feedback to the user regarding the position of the light source in the three-dimensional environment and the spatial relationship between the viewpoint of the user, the simulated object and the light source, which improves the situational awareness of the user and reduces cognitive imbalance between the simulated object and the displayed environment, which provides improved feedback to the user.

In some embodiments, the light in the three-dimensional environment changes (e.g., changes in color, brightness, position, and/or orientation due to changes in illumination in the physical environment represented in the three-dimensional environment and/or changes in simulated illumination in the three-dimensional environment). In some embodiments, in response to a change in light in a three-dimensional environment, the computer system: displaying a change in view of the three-dimensional environment according to the change in light; and altering an appearance of the one or more edges of the simulated three-dimensional object to display a visual effect of reflecting the altered light from the one or more edges based on the properties of the respective simulated material. For example, as described herein with reference to fig. 7L-7M, the appearance and location of specular reflection from the user interface element 7054 changes according to changes in the light source 7056 and/or the light source 7058. Displaying a change in specular reflection from the one or more edges of the simulated three-dimensional object in response to a change in light in the three-dimensional environment, wherein the light is based on simulated illumination and/or physical light in the user's surrounding physical environment, gives the simulated object an appearance of how the physical object interacts with the light, and provides visual feedback to the user regarding the position of the light source in the three-dimensional environment and the spatial relationship between the light source, the simulated object, and the user's point of view, which improves the user's situational awareness and reduces cognitive imbalance between the simulated object and the display environment, which provides improved feedback to the user.

In some embodiments, the second region of the simulated three-dimensional object is displayed as having an appearance (e.g., a second etched region) that removes a portion of the corresponding simulated material from the surface of the simulated three-dimensional object; and the second user interface element includes a second region simulating the three-dimensional object (e.g., the second region simulating the three-dimensional object identifies the second user interface element); the second region of the simulated three-dimensional object is a second user interface element). In some embodiments, the computer system detects an input directed to a second user interface element; and in response to detecting an input directed to the second user interface element: a respective operation associated with the second user interface element is performed in the user interface. In some implementations, the simulated three-dimensional object includes a plurality of etched regions, each etched region corresponding to a respective user interface element, and the user may direct input to the respective user interface element to perform a respective associated operation in the user interface. For example, as described herein with reference to fig. 7L-7O, the user interface element 7054 includes a plurality of recessed regions 7054-b and 7054-c configured to perform respective operations in a user interface. Displaying multiple regions of the simulated three-dimensional object as portions of material removed from the surface of the object and associating those regions with user interface elements provides additional regions with which a user can easily interact to perform operations in the user interface, and distinguishes those regions from other regions along the surface of the simulated object to provide visual feedback to the user that those regions are interactive, which provides additional control options that reduce the number of inputs required to access the associated operations and provides improved feedback to the user.

In some embodiments, a computer system receives an input corresponding to an input text; and displaying the entered text in the first user interface element. In some embodiments, the first user interface element is or includes a text input field. In some implementations, the entered text is displayed in a text input field of the first user interface element (e.g., when the first user interface element has focus). For example, as described herein with reference to fig. 7L, the recessed areas 7054-b and 7054-c of the user interface element 7054 are optionally content input areas, such as text input boxes. The method may further include setting up an area of the simulated three-dimensional object displayed as a portion of material removed from a surface of the simulated object as a text input area configured to receive text input, distinguishing the area from other areas along the surface of the simulated object to provide visual feedback to the user that the area is interactive, and in some embodiments providing visual feedback to the user that the area is a text input area rather than an activatable control area, which provides improved feedback to the user.

In some implementations, the computer system detects a second input corresponding to a user focusing on a focus (e.g., attention) on the first user interface element before detecting the first input corresponding to activation of the first user interface element. In some cases, the second input and the first input are consecutive portions of the same input. In some implementations, in response to detecting a second input corresponding to a user focusing on the first user interface element: in accordance with a determination that the first user interface element is a first type of user interface element (e.g., a button or activatable control), the computer system displays a first region of the simulated three-dimensional object as having an appearance that a portion of the respective simulated material protrudes from a surface of the simulated three-dimensional object (e.g., a protruding portion of the respective simulated material, a portion that extends or protrudes from the surface of the simulated three-dimensional object, or a portion of the respective simulated material that is positioned in a manner that is separated from the surface of the simulated three-dimensional object to some extent); and in accordance with a determination that the first user interface element is a second type of user interface element (e.g., a text input field) that is different from the first type, the computer system maintains the display of the first region of the simulated three-dimensional object as having an appearance that removes a portion of the corresponding simulated material from the surface of the simulated three-dimensional object.

For example, as described herein with reference to fig. 7L-7O, some types of user interface elements are displayed as raised areas (e.g., some recessed areas are changed to raised areas, and/or some raised areas are even further raised) while focus (e.g., the user's attention) is focused on the user interface element. Displaying a region of the simulated three-dimensional object as having a portion of material removed from a surface of the simulated object and changing an appearance of the region to an appearance that has a portion of material raised from the surface of the simulated object if the region is associated with a user interface element of a first type and maintaining the appearance of the region as having material removed if the region is associated with a user interface element of a second type provides visual feedback to a user regarding a type of user interface element associated with the region and thus a type of user interface operation configured to be performed in response to further interaction with the user interface element of the region, which provides improved feedback to the user.

In some embodiments, the third region of the simulated three-dimensional object is displayed as having an appearance that bulges a portion of the respective simulated material from the surface of the simulated three-dimensional object (e.g., a bulge or protrusion of the respective simulated material of the simulated three-dimensional object, as a portion of the simulated material layered on the surface of the simulated three-dimensional object, as an embossed or extruded region, or as a portion of the respective simulated material positioned in a manner separate from the surface of the simulated three-dimensional object, or other similar visual effect); and the third user interface element includes a third region of the simulated three-dimensional object (e.g., the third region of the simulated three-dimensional object identifies the third user interface element; the third region of the simulated three-dimensional object is the third user interface element). In some embodiments, the computer system: detecting an input directed to a third user interface element; and in response to detecting an input directed to the third user interface element: a respective operation associated with the third user interface element is performed in the user interface. For example, as described herein with reference to fig. 7L-7O, the raised regions 7054-d of the user interface element 7054 are optionally layered on top of the user interface element 7054 or placed on top of the user interface element.

In some embodiments, a third user interface element comprising a third region of the simulated three-dimensional object is raised away from the surface of the simulated three-dimensional object in response to a first portion of input directed to the third user interface element and a corresponding operation associated with the third user interface element is performed in response to a second subsequent portion of input directed to the third user interface element, as described in more detail with reference to method 12000 (e.g., the first portion of input involves a user focusing attention on the third user interface element and the second portion of input involves a user performing a selection gesture (such as an air gesture) directed to the third user interface element, optionally while the user's attention remains focused on the third user interface element). In some implementations, the displaying the third user interface element as protruding or lifting from the surface of the simulated three-dimensional object is performed in accordance with a determination that the third user interface element is a first type of user interface element (e.g., a button or an activatable control). In some implementations, the third user interface element is raised a first amount in response to a first type of input (e.g., detecting that the user is looking at the third user interface element and/or that the user's hand is in a predefined ready state), and a second, different amount in response to a second, different type of input (e.g., detecting that the user's hand is in a predefined ready state and/or that a predefined gesture (e.g., an air gesture) is being performed), as described in more detail herein with reference to method 12000. In some embodiments, the simulated three-dimensional object includes any number of etched regions and/or any number of raised regions, each etched region optionally corresponding to a respective user interface element, each raised region optionally corresponding to a respective user interface element, and the user may direct input to a respective user interface element to perform a respective associated operation in the user interface.

Displaying an area of the simulated three-dimensional object as a portion of material protruding from a surface of the simulated object and associating the area with the user interface element distinguishes the area from other areas along the surface of the simulated object and provides visual feedback to a user that can interact with the protruding area to perform operations in the user interface, and distinguishes a type of user interface operation associated with the protruding area from a type of user interface operation associated with the recessed area (which is displayed as having the appearance of removing a portion of material from the surface of the user interface element), which provides improved feedback to the user.

In some implementations, the content of the simulated three-dimensional object (e.g., application content) is displayed in a second portion of the simulated three-dimensional object having a simulated thickness behind a front surface of the simulated three-dimensional object, where the front surface of the simulated three-dimensional object corresponds to a portion of an outer surface of the simulated three-dimensional object that faces a user viewpoint (e.g., as described herein with reference to content displayed as embedded in user interface element 7050 and/or content displayed as embedded in user interface element 7052). In some embodiments, the content of the simulated three-dimensional object is displayed as embedded in a corresponding simulated material of the simulated three-dimensional object. In some embodiments, the content of the simulated three-dimensional object is displayed along a back surface of the simulated three-dimensional object (e.g., a viewpoint opposite and facing away from the user's viewpoint). Displaying content embedded within a simulated three-dimensional object (e.g., between two surfaces of the simulated three-dimensional object), simulating the appearance of an object made of transparent or partially transparent material, and distinguishing the content from other user interface elements (such as content input areas or activatable controls displayed as concave or convex areas relative to the surface of the simulated object) and from other user interface elements that are not interactive and/or that are not representations of physical objects in the physical environment surrounding the user provides improved feedback to the user, and improves the situational awareness of the user, which improves user safety by helping the user avoid collisions with physical objects in the physical space, and reduces the likelihood that the user will suffer from motion sickness when using the computer system.

In some implementations, a simulated three-dimensional object having a first set of characteristics is displayed at a first location relative to a viewpoint of a user at a first time. In some embodiments, the first set of characteristics includes characteristics other than location, such as depth, thickness, brightness, color, and/or color temperature. In some embodiments, after the first time, the simulated three-dimensional object is moved to a second position relative to the viewpoint of the user. In some embodiments, the computer system displays, at a second location, the simulated three-dimensional object having a second set of characteristics different from the first set of characteristics according to the changed viewpoint of the user in response to a change in the location of the simulated three-dimensional object relative to the viewpoint of the user.

In some embodiments, the thickness of the simulated three-dimensional object is changed in response to a change in the position of the simulated three-dimensional object relative to the viewpoint of the user. Thus, in some embodiments, the appearance of one or more edges of the simulated three-dimensional object is changed to show the visual effect of reflecting light from the one or more edges based on the changed thickness of the simulated three-dimensional object (e.g., corresponding to reflecting light from a thicker edge if the thickness of the simulated three-dimensional object increases or reflecting light from a thinner edge if the thickness of the simulated three-dimensional object decreases). For example, as described herein with reference to fig. 7L-7M, the increase in thickness of the user interface element 7054 in fig. 7M exceeds a proportional increase in thickness associated with the movement of the user interface element 7054 from its position farther from the user's viewpoint in fig. 7L to its position closer to the user's viewpoint in fig. 7M.

In the physical world, as the spatial relationship between a physical object and a user's point of view changes (e.g., the object and the user's point of view move relative to each other), the object appears to the user to be proportionally larger when closer to the user and proportionally smaller when further from the user, with the size of the object remaining the same. In a computer-generated three-dimensional environment, a computer-generated (e.g., simulated) object may be displayed having dimensions independent of the spatial relationship between the simulated object and the viewpoint of the user, such as the object being displayed with a change in one or more dimensions that exceeds a proportional change associated with the spatial movement. The independence and magnification of the displayed visual change in size of the object from the proportional change associated with the spatial movement enhances the appearance of the user interface object as a computer-generated simulated object and distinguishes the simulated object from other displayed representations of objects corresponding to physical objects in the user's surrounding physical environment, which provides improved feedback to the user and improves the user's situational awareness, which increases user safety by helping the user avoid collisions with physical objects in the physical space and reduces the likelihood that the user will suffer from motion sickness when using the computer system.

It should be understood that the particular order in which the operations in fig. 11 are described is merely an example and is not intended to suggest that the order is the only order in which the operations may be performed. Those of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein with respect to other methods described herein (e.g., methods 8000, 9000, 10000, 12000, 13000, and 14000) apply in a similar manner to method 11000 described above with respect to fig. 11. For example, the user interface elements or objects (including their appearance and regions), the physical objects and/or representations of physical objects, virtual objects, user viewpoints, movements, three-dimensional environments (including their views and appearances), display generating components, light sources, reflections, and/or shadows described above with reference to method 11000 optionally have one or more of the characteristics of the user interface elements or objects (including their appearances and regions), the physical objects and/or representations of physical objects, virtual objects, user viewpoints, movements, three-dimensional environments (including their views and appearances), display generating components, light sources, reflections, and/or shadows described herein with reference to other methods described herein (e.g., methods 8000, 9000, 10000, 12000, 13000, and 14000). For the sake of brevity, these details are not repeated here.

FIG. 12 is a flow diagram of a method 12000 of separating layers of user interface elements to different extents in response to different user interactions including a user focusing attention to the user interface elements, according to some embodiments.

In some embodiments, the method 12000 is performed at a computer system (e.g., computer system 101 in fig. 1) that includes or communicates with a display generation component (e.g., display generation component 120 in fig. 1,3, and 4) (e.g., heads-up display, touch screen, and/or projector) and one or more input devices such as one or more cameras (e.g., a camera directed downward toward a user's hand (e.g., color sensor, infrared sensor, and/or other depth sensing camera) or directed forward from the user's head). In some embodiments, the method 12000 is managed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors of a computer system, such as one or more processors 202 of the computer system 101 (e.g., the control unit 110 in fig. 1A). Some operations in method 12000 are optionally combined, and/or the order of some operations is optionally changed.

In some embodiments, method 12000 is performed at a computer system (e.g., computer system 101 in fig. 1, computer system 101 described with respect to fig. 7A-7G, 7H-7J, 7K-7O, 7P, 7Q-7W, and/or 7X-7Z). In some implementations, the computer system communicates with a display generating component (e.g., the first display generating component is a heads-up display, a head-mounted display (HMD), a display, a touch screen, a projector, etc.) and optionally one or more input devices (e.g., a camera, a controller, a touch-sensitive surface, a joystick, buttons, a glove, a watch, a motion sensor, and/or an orientation sensor). In some embodiments, the display generating component is the display generating component 7100 described with respect to fig. 7A-7Z. In some embodiments, the computer system is an integrated device having one or more processors and memory and at least some of the one or more input devices enclosed in the same housing as the display generating component. In some embodiments, the computer system includes a computing component (e.g., a server, a mobile electronic device such as a smart phone or tablet device, a wearable device such as a watch, wristband, or earpiece, a desktop computer, or a laptop computer) that includes one or more processors and memory separate from the display generating component and/or the one or more input devices. In some embodiments, the display generating component and the one or more input devices are integrated and packaged in the same housing. Many of the features of method 12000 are described with respect to fig. 7P, according to some embodiments.

In method 12000, the computer system displays (12002), via a display generation component, a user interface comprising a view of the three-dimensional environment and user interface elements associated with respective surfaces in the three-dimensional environment (e.g., a front surface of a user interface tray (such as user interface element 7054 in fig. 7L), or a common plane on which to display icon group 7070 as shown in fig. 7P). A first set of one or more portions of the user interface element are associated with (e.g., included in) a first layer of the user interface element (12004); a second set of one or more portions of the user interface element is associated with a second layer of the user interface element; and the user interface element is displayed having an appearance indicating a first degree of separation between the first layer and the second layer. In some embodiments, the first degree of separation is zero when the user is not focusing on the user interface element.

In method 12000, the computer system detects (12006) a first input corresponding to the user focusing attention on the user interface element (e.g., detects an input corresponding to the user focusing attention on the user interface element, such as detecting that the user's gaze is directed toward the user interface element and/or that a cursor is moving on the user interface element). In response to detecting a first input (12008) corresponding to a user focusing attention on a user interface element: the computer system changes (12010) the appearance of the user interface element to indicate a second degree of separation between the first layer and the second layer. The second degree is different (e.g., greater) than the first degree. In some implementations, in response to detecting the first input, an appearance of the user interface element is changed to indicate a different degree of separation between one or more layers of the user interface element and a corresponding surface in the three-dimensional environment. Displaying a user interface element associated with a surface in the displayed three-dimensional environment and comprising a plurality of layered elements and varying a degree of separation between the plurality of layered elements in response to a user concentrating attention to the user interface element provides visual feedback to the user as to which object in the user interface has focus and may be selected or activated in response to further user input, which is particularly helpful in cases where the user input does not involve direct physical contact with the user interface element (e.g., via touch input on a touch screen), which provides improved feedback to the user.

In some implementations, the user interface element can be activated to perform a corresponding operation in the user interface that is different from changing the appearance of the user interface element (e.g., different from changing the degree of separation between different layers of the user interface element). In some embodiments, in response to detecting a first input corresponding to a user focusing on a user interface element, changing an appearance of the user interface element without (however) performing a corresponding operation (e.g., requiring additional interaction in order to perform the corresponding operation) (e.g., changing the appearance of the user interface element provides visual feedback indicative of interaction with the user interface element and other than user interface operations performed in response to interaction with the user interface element). For example, as described herein with reference to fig. 7P, in response to the user 7002 gazing at the icon 7060, the degree of separation between the layers 7060-1 through 7060-3 is changed, but activation of the icon 7060 (e.g., to launch an application, open a file, start a communication session, or other associated operation) is not performed until a selection gesture directed to the icon 7060 (e.g., an air gesture such as pinching while the user's gaze is directed to the icon, or a tap at a location corresponding to the icon in a three-dimensional environment) is performed. Configuring the user interface element to be activatable to perform one or more operations in the user interface (the appearance of the user interface element being changed to indicate a different degree of separation between its plurality of layered elements in response to the user concentrating attention to the user interface element) provides visual feedback to the user as to which object in the user interface has focus and will trigger performance of the one or more operations in the user interface in response to further user input, which provides improved feedback to the user and reduces the amount of input required to perform the operation.

In some implementations, detecting the first input includes detecting that the user is looking at (e.g., looking toward) the user interface element (e.g., and optionally, detecting that the user's hand is not in a predefined ready state, or regardless of whether the user's hand is in or moving to a predefined ready state). For example, as described herein with reference to fig. 7P, in response to the user 7002 gazing at the icon 7060 as shown in the scene 7064 (e.g., the hand 7020 is not in a ready state), the degree of separation between the layers 7060-1 to 7060-3 is changed to the degree of separation shown in the view 7076. Configuring the user interface element to be capable of being activated to perform one or more operations in the user interface in response to a user gazing at the user interface element, and changing the appearance of the user interface element accordingly to indicate different degrees of separation between its plurality of layered elements, provides an intuitive, simple and quick mechanism for interacting with the user interface, which provides improved feedback to the user and reduces the number of inputs required to perform the operations.

In some implementations, detecting the first input includes detecting that the user's hand is in a predefined ready state (e.g., and optionally, detecting that the user is not looking at the user interface element, or regardless of whether the user is looking at the user interface element). In some embodiments, the user's hand is in a predefined ready state if the user's hand is raised (e.g., rather than being lowered at the user's hand, and optionally if the user's hand is raised in a curved elbow rather than with the arms of the hand extending) and/or in a configuration indicating that a gesture is ready to be performed (such as palm down and index finger extending and/or lifting, or palm inward (e.g., toward the user's midline) and thumb resting on index finger, or other predefined hand configuration). For example, as described herein with reference to fig. 7P, in response to the user 7002 looking at the icon 7060 while the hand 7020 is in a ready state as shown in the scene 7066, the degree of separation between the layers 7060-1 to 7060-3 is changed to the degree of separation shown in the view 7078. Configuring the user interface element to be capable of being activated to perform one or more operations in the user interface in response to the user's hand being in a predefined ready state, and changing the appearance of the user interface element accordingly to indicate different degrees of separation between its plurality of layered elements, provides an intuitive, simple and quick mechanism for interacting with the user interface, which provides improved feedback to the user and reduces the number of inputs required to perform the operations.

In some implementations, detecting the first input includes detecting that the user is looking at the user interface element and that the user's hand is in a predefined ready state. For example, as described herein with reference to fig. 7P, in response to the user 7002 looking at the icon 7060 while the hand 7020 is in a ready state as shown in the scene 7066, the degree of separation between the layers 7060-1 to 7060-3 is changed to the degree of separation shown in the view 7078. Configuring the user interface element to be capable of being activated to perform one or more operations in the user interface in response to the user gazing at the user interface element while the user's hand is in a predefined ready state, and changing the appearance of the user interface element accordingly to indicate different degrees of separation between its plurality of layered elements, provides an intuitive, simple and quick mechanism for interacting with the user interface, which provides improved feedback to the user and reduces the number of inputs required to perform the operations.

In some implementations, in accordance with a determination that the input is a first type of input that includes a user's gaze toward the user interface element (e.g., the user's hand is not in or moving to a predefined ready state), the second degree of separation has a first magnitude (which is different from the magnitude of the first degree of separation). In some implementations, in accordance with a determination that the input is a second type of input that includes a user's gaze directed to the user interface element and the user's hand being in a predefined ready state, the second degree of separation has a second magnitude that is different from the first magnitude (and also different from the magnitude of the first degree of separation). In some implementations, in response to an input corresponding to a user directing the user's gaze at the user interface element while the user's hand (e.g., has been) in a predefined ready state, the appearance of the user interface element changes from indicating a first degree of separation to indicating a second degree of separation of a second magnitude, with or without an intermediate display displaying the user interface element as having an appearance indicating the second degree of separation of the first magnitude. In some implementations, the appearance of the user interface element is changed from indicating a first degree of separation to indicating a second degree of separation of the first magnitude in response to a first respective input (or a first portion of the respective input) corresponding to the user directing the user's gaze at the user interface element while the user's hand is not in the predefined ready state. In some implementations, in response to a second corresponding input (or a second portion of the corresponding input) corresponding to a user's hand moving to a predefined ready state while the user's gaze is directed toward the user interface element, the appearance of the user interface element is changed from (e.g., further) indicating a second degree of separation of the first magnitude to indicating a second degree of separation of the second magnitude. For example, as described herein with reference to fig. 7P, in response to the user 7002 looking at the icon 7060 and the hand 7020 not being in the ready state as shown in the scene 7064, the degree of separation between the layers 7060-1 to 7060-3 is changed to the degree of separation shown in the view 7076, and in response to the user 7002 looking at the icon 7060 and the hand 7020 being in the ready state as shown in the scene 7066, the degree of separation between the layers 7060-1 to 7060-3 is changed to the degree of separation shown in the view 7078. Indicating different degrees of separation between multiple layered elements of the user interface element in response to different types of inputs directed to the user interface element, and in particular in response to gaze input vs gaze input plus a user's hand being in a predefined ready state, enables disambiguation between the different inputs and associated operations to be performed in the user interface, and provides visual feedback to the user as to which type of input has been detected and which operation is to be performed in the user interface in response, which provides improved feedback to the user and provides additional control options without cluttering the user interface with additional display controls.

In some implementations, after detecting the first input, when the appearance of the user interface element indicates a second degree of separation between the first layer and the second layer, the computer system detects a second input corresponding to a selection of the user interface element (e.g., a selection gesture, such as a pinch while the user's gaze is directed at the user interface element, or a tap at a location corresponding to the user interface element in a three-dimensional environment). In some implementations, in response to detecting the second input, the computer system changes an appearance of the user interface element to indicate a third degree of separation between the first layer and the second layer. In some embodiments, the third degree of separation is less than the second degree of separation. In some implementations, responsive to detecting an end of the second input (e.g., a subsequent portion of the second input, such as a release gesture), a respective operation in the user interface that is different from changing an appearance of the user interface element is performed. For example, as described herein with reference to fig. 7P, in response to the user 7002 performing a selection gesture as described with reference to the scene 7066 (e.g., while looking at the icon 7060), the degree of separation between the layers 7060-1 through 7060-3 decreases from that shown in view 7078, and in response to completion of the selection gesture, an operation associated with the icon 7060 is performed in the environment 7104. In response to further input directed to the user interface element after focusing on the user interface element, the degree of separation between layers of the user interface element having focus is reduced, providing visual feedback to the user that the user interface element having focus has been selected for further interaction, which is particularly useful where the user input does not involve direct physical contact with the user interface element (e.g., via touch input on a touch screen), which provides improved feedback to the user.

In some implementations, the second input includes an progression of input that performs a predefined gesture. In some implementations, the predefined gesture is a pinch in which two fingers move toward each other. In some implementations, the predefined gesture is a flick gesture, for example, at a location on the touch-sensitive surface corresponding to the displayed user interface element. Generally, in implementations where the predefined gesture is a gesture performed in the air (e.g., in the absence of physical contact with an input device, the input device provides information to the computer system as to which user interface element is the target of user input, such as contact with a user interface element displayed on a touch screen, or contact with a mouse or touchpad to move a cursor to the user interface element), the gesture considers the user's gaze to determine the target of user input. Thus, in implementations involving air gestures, the predefined gesture is, for example, an air flick gesture, e.g., a combination of gaze toward a user interface element and (e.g., concurrent with) the following: the movement of the user's hand fingers toward the user interface element, the movement of the user's hand fingers toward the user interface element (optionally wherein the user's hand fingers stretch toward the user interface element), the downward movement of the user's hand fingers (e.g., mimicking a mouse click motion or a tap on a touch screen), or other predefined movements.

In some implementations, as the predefined gesture is made (e.g., the input is made such as by moving two fingers toward each other in a pinch gesture, or moving a finger toward the user interface element in a flick gesture), the computer system (e.g., gradually) makes the appearance of the user interface element change toward the indication of the third degree of separation by an amount corresponding to the amount of input made (e.g., reduces the degree of separation between the first layer and the second layer). In some implementations, as the predefined gesture is reversed (e.g., the progress of the input reverses the predefined gesture, such as reversing the pinch gesture by moving two fingers away from each other, or reversing the flick gesture by moving the fingers away from the user interface element), the computer system reverses (e.g., gradually) the appearance of the user interface element toward a change indicative of the second degree of separation by an amount corresponding to the progress of the input (e.g., increases the degree of separation between the first layer and the second layer). For example, as described herein with reference to fig. 7P, in response to the progression of a selection gesture as described with reference to scene 7066, the degree of separation between layers 7060-1 to 7060-3 is changed, however the change in the degree of separation is reversed in response to the reversal of the selection gesture before the selection gesture is completed. In response to further input of the user interface element directed to the predefined gesture after focusing on the user interface element, a decrease in the degree of separation between layers of the user interface element having focus is progressively made and the decrease in the degree of separation is made reversible in response to reversal of the predefined gesture, providing visual feedback to the user that the user interface element having focus has been selected for further interaction, and providing the user with greater control over whether to make or cancel interaction with the selection, providing the user with improved feedback and providing additional control options without cluttering the user interface with additional display controls.

In some embodiments, the computer system displays the second set of one or more portions of the user interface element as having a visual effect that the first set of one or more portions of the user interface element associated with the first layer projects light onto the second layer. In some embodiments, a first set of one or more portions of the user interface element are displayed having an appearance that simulates the light projected onto the first layer by a second set of one or more portions of the user interface element associated with the second layer. For example, as described herein with reference to fig. 7P, the layers 7060-1 to 7060-3 are optionally displayed to project light on each other. The visual effect of the respective layers of the display user interface element projecting light onto another layer of the user interface element and/or the different layers projecting light onto each other enhances the appearance of the layers of the user interface element being separated from each other, which enhances the visual feedback provided to the user identifying the object with which the user is interacting, which provides improved feedback to the user.

In some embodiments, the visual effect simulates a first set of one or more portions of the user interface element reflecting light (e.g., from a different light source in the three-dimensional environment than the first set of one or more portions of the user interface element) onto a second set of one or more portions of the user interface element. For example, as described herein with reference to fig. 7P, the layers 7060-1 to 7060-3 are optionally displayed to reflect light from one or more individual (e.g., physical and/or analog) light sources in the environment 7104 onto each other. The visual effect of the respective layers of the display user interface element reflecting light onto another layer of the user interface element and/or the different layers reflecting light onto each other enhances the appearance of the layers of the user interface element being separated from each other, which enhances the visual feedback provided to the user identifying the object with which the user is interacting, which provides improved feedback to the user.

In some implementations, the visual effect simulates the first set of one or more portions of the user interface element emitting light onto the second set of one or more portions of the user interface element (e.g., the first set of one or more portions of the user interface element acts as a light source). For example, as described herein with reference to fig. 7P, the layers 7060-1 to 7060-3 are optionally displayed to emit light onto each other. The visual effect of the respective layers of the display user interface element emitting light onto another layer of the user interface element and/or the different layers emitting light onto each other enhances the appearance of the layers of the user interface element being separated from each other, which enhances the visual feedback provided to the user identifying the object with which the user is interacting, which provides improved feedback to the user.

In some embodiments, the computer system displays a second set of one or more portions of the user interface element as having an appearance that simulates the first set of one or more portions of the user interface element associated with the first layer casting one or more shadows onto the second layer. For example, as described herein with reference to fig. 7P, the layers 7060-1 to 7060-3 are optionally displayed to transmit shadows onto each other. The visual effect of the respective layers displaying the user interface element casting shadows onto another layer of the user interface element and/or the different layers casting shadows onto each other enhances the appearance of the layers of the user interface element being separated from each other, which enhances visual feedback provided to the user identifying the object with which the user is interacting, which provides improved feedback to the user.

In some embodiments, in response to detecting the first input, the computer system displays an animation of one or more portions of the user interface element (e.g., in addition to separating layers of the user interface element). In some implementations, the animation is performed in response to the performance of the input of the predefined gesture, and the animation is reversed in response to the performance of the input of the reversed predefined gesture, as described in further detail herein with reference to method 12000. For example, as described herein with reference to fig. 7P, in some embodiments, the icon 7060 is animated in response to focusing attention on the icon 7060 and/or performing an input of a selection gesture directed to the icon. In conjunction with varying the degree of separation between the plurality of layered elements of the user interface element, displaying an animation of at least a portion of the user interface element in response to the user focusing attention on the user interface element enhances visual feedback provided to the user identifying an object with which the user is interacting, which provides improved feedback to the user.

In some embodiments, displaying an animation of the one or more portions of the user interface element includes rotating the one or more portions of the user interface element (e.g., rotating or rotating a graphic such as an icon or glyph, such as rotating or rotating an envelope graphic of icon 7060 in fig. 7P). In conjunction with varying the degree of separation between the plurality of layered elements of the user interface element, displaying an animation of at least a portion of the rotation of the user interface element in response to the user focusing attention on the user interface element, visual feedback provided to the user identifying an object with which the user is interacting is enhanced, which provides improved feedback to the user.

In some embodiments, displaying the animation of the one or more portions of the user interface element includes sliding the one or more portions of the user interface element in a respective direction (e.g., sliding along or relative to another portion of the user interface element, such as moving a slider thumb along a slider, or sliding the envelope graphic of the icon 7060 past the circular element of the icon 7060 in fig. 7P). In conjunction with varying the degree of separation between the plurality of layered elements of the user interface element, displaying an animation of at least a portion of the user interface element sliding in a corresponding direction in response to the user focusing attention on the user interface element enhances visual feedback provided to the user identifying an object with which the user is interacting, which provides improved feedback to the user.

In some embodiments, displaying the animation of the one or more portions of the user interface element includes changing the scale of the one or more elements of the displayed graphic, followed by reversing the change in scale of the one or more elements of the displayed graphic (e.g., increasing the scale, followed by decreasing the scale, and/or decreasing the scale, followed by increasing the scale, such as expanding, followed by contracting, and/or contracting, followed by expanding, for example, to represent an inhalation then exhalation-type effect, or vice versa). Using the example from fig. 7P, the animation of the icon 7060 optionally includes continuously zooming in and out, and/or rotating or turning envelope graphics and/or circular elements of the icon 7060. In some implementations, displaying the animation includes changing the scale of the displayed graphic and reversing the change in scale. In some implementations, displaying the animation includes changing the scale of the plurality of elements comprising the displayed graphic and changing the scale of the inverse while optionally moving (e.g., translating) and/or rotating the elements of the graphic relative to each other (or moving and/or rotating the graphic as a whole). In some embodiments, displaying the animation includes repeating a portion of the animation (e.g., repeatedly increasing and decreasing the scale, for example, to repeatedly represent an inhalation then exhalation effect). In conjunction with varying the degree of separation between the plurality of layered elements of the user interface element, displaying an animation of at least a portion of the user interface element continuously increasing or decreasing in scale in response to the user focusing attention to the user interface element enhances visual feedback provided to the user identifying an object with which the user is interacting, which provides improved feedback to the user.

In some embodiments, displaying an animation of the one or more portions of the user interface element includes displaying a preview of content associated with the user interface element (e.g., the user interface element is a file, such as a document, an image, a video, or an email, and the animation includes displaying a preview of the file and/or information about the file or metadata for the file, the user interface element is a folder of files, and the animation includes displaying a preview of the files in the folder, or the user interface element is an application icon that can be activated to launch an application, and the animation includes displaying a user interface of the application). For example, as shown in fig. 7P, the icon 7060 includes an envelope graphic (e.g., representing an email), and the animation optionally includes displaying a preview of the email associated with the icon 7060. In some embodiments, the preview is displayed over a portion of the displayed user interface (e.g., less than all of the area of the user interface, or less than substantially all of the area of the user interface, such as less than 90%, 80%, 70%, 60%, 50%, or any other amount between 10% and 100% of the user interface). In conjunction with varying the degree of separation between the plurality of layered elements of the user interface element, displaying an animation that shows a preview of content associated with the user interface element in response to the user focusing attention on the user interface element, visual feedback provided to the user identifying an object with which the user is interacting is enhanced, and an indication of the type of operation configured to be performed in response to further interaction with the user interface element is provided, which provides improved feedback to the user.

In some embodiments, the user interface element includes a representation of an envelope (e.g., an envelope icon or graphic, optionally associated with a communication application or communication file (such as an email), as shown for example by icon 7060 in fig. 7P), and displaying an animation of the one or more portions of the user interface element includes displaying an animation of opening the envelope. In conjunction with varying the degree of separation between the plurality of layered elements of the user interface element, displaying an animation that shows a preview of content associated with the user interface element in response to the user focusing attention on the user interface element, visual feedback provided to the user identifying an object with which the user is interacting is enhanced, and an indication of the type of operation configured to be performed in response to further interaction with the user interface element is provided, which provides improved feedback to the user.

It should be understood that the particular order in which the operations in fig. 12 are described is merely an example and is not intended to suggest that the order is the only order in which the operations may be performed. Those of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein with respect to other methods described herein (e.g., methods 8000, 9000, 10000, 11000, 13000, and 14000) apply in a similar manner to method 12000 described above with respect to fig. 12. For example, the user interface elements or objects (including their appearance and regions), virtual objects, layers, degrees or amounts of separation, user viewpoints, movements, three-dimensional environments (including their views and appearances), display generating components, inputs, gestures, animations, user interface operations, light sources, reflections, and/or shadows described above with reference to method 12000 optionally have one or more of the characteristics of the user interface elements or objects (including their appearances and regions), virtual objects, layers, degrees or amounts of separation, user viewpoints, movements, three-dimensional environments (including their views and appearances), display generating components, inputs, gestures, animations, user interface operations, light sources, reflections, and/or shadows described herein with reference to other methods described herein (e.g., methods 8000, 9000, 10000, 11000, 13000, and 14000). For the sake of brevity, these details are not repeated here.

Fig. 13 is a flow chart of a method 13000 of changing an appearance of a selectable user interface object in response to an input corresponding to the selectable user interface object and indicating that it is ready to interact with the selectable user interface object, according to some embodiments.

In some embodiments, the method 13000 is performed at a computer system (e.g., computer system 101 in fig. 1) that communicates with a display generating component (e.g., display generating component 120 in fig. 1, 3, and 4, such as display 7100 in fig. 7A-7Z) (e.g., heads-up display, touch screen, and/or projector) and optionally one or more input devices such as one or more cameras (e.g., a camera directed downward toward the user's hand (e.g., color sensor, infrared sensor, and/or other depth sensing camera) or directed forward from the user's head). In some embodiments, method 13000 is managed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors of a computer system, such as one or more processors 202 of computer system 101 (e.g., control unit 110 in fig. 1A). Some operations in method 13000 are optionally combined and/or the order of some operations is optionally changed.

In some embodiments, method 13000 is performed at a computer system (e.g., computer system 101 in fig. 1, computer system 101 described with respect to fig. 7A-7G, 7H-7J, 7K-7O, 7P, 7Q-7W, and/or 7X-7Z). In some implementations, the computer system communicates with a display generating component (e.g., the first display generating component is a heads-up display, a head-mounted display (HMD), a display, a touch screen, a projector, etc.) and optionally one or more input devices (e.g., a camera, a controller, a touch-sensitive surface, a joystick, buttons, a glove, a watch, a motion sensor, and/or an orientation sensor). In some embodiments, the display generating component is the display generating component 7100 described with respect to fig. 7A-7Z. In some embodiments, the computer system is an integrated device having one or more processors and memory and at least some of the one or more input devices enclosed in the same housing as the display generating component. In some embodiments, the computer system includes a computing component (e.g., a server, a mobile electronic device such as a smart phone or tablet device, a wearable device such as a watch, wristband, earpiece, desktop computer, or laptop computer) that includes one or more processors and memory separate from the display generating component and/or one or more input devices. In some embodiments, the display generating component and the one or more input devices are integrated and packaged in the same housing. According to some embodiments, many of the features of method 13000 are described with respect to fig. 7Q-7W.

In method 13000, a computer system displays (13002) a user interface object via a display generation component while a first view of a three-dimensional environment (e.g., environment 7104 (fig. 7Q)) is visible (e.g., via the display generation component). The user interface object (e.g., user interface object 7080 (fig. 7Q)) includes a first selectable object (e.g., any of regions 7080-b, 7080-c, or 7080-d (fig. 7Q)) that can be activated to perform a respective operation in a three-dimensional environment. In some embodiments, the first selectable object occupies a first region of the user interface object (e.g., any of regions 7080-b, 7080-c, or 7080-d (fig. 7Q)), and a surface of the first region of the user interface object (e.g., a surface of the first selectable object) is displayed with a first degree of separation from a surface of a second region of the user interface object (e.g., region 7080-a (fig. 7Q)), wherein the second region is different from (e.g., adjacent to) the first region. In some embodiments, the first degree of separation is zero (e.g., there is no separation; the surface of the first region is flush with the surface of the second region).

In method 13000, a computer system detects (13004) a first input corresponding to a first selectable object (e.g., a first input indicating that a user is ready to interact with the first selectable object). In response to detecting a first input (13006) corresponding to a first selectable object: in accordance with a determination that the first input indicates that the user is ready to interact with the first selectable object, the computer system changes (13008) (e.g., increases) the thickness of the first selectable object (e.g., does not perform a corresponding operation in a three-dimensional environment). For example, as shown in fig. 7R and 7T and described with reference to these figures, in response to detecting that the user 7002 is focused on the region 7080-b, the thickness of the region 7080-b is increased. In another example, as shown in FIG. 7S and described with reference thereto, in response to detecting that the user 7002 is focused on the region 7080-c, the thickness of the region 7080-c is increased. In some embodiments, changing the thickness of the first selectable object of the user interface object (e.g., changing from the first thickness to the second thickness) in response to detecting the first input indicating that the user is ready to interact with the first selectable object includes (or the change is performed and also plus): the surface of the first selectable object is displayed as having a second degree of separation from the surface of the second region of the user interface object, wherein the second degree of separation is different (e.g., greater) than the first degree of separation. In some embodiments, the first thickness is zero (e.g., the first selectable object is flush with the surface of the user interface object) (e.g., as in the case where the regions 7080-b, 7080-c, and 7080-d in fig. 7Q are flush with the surface of the user interface object 7080). In some embodiments, in accordance with a determination that the first input does not indicate that the user is ready to interact with the first selectable object, the computer system does not change the thickness of the first selectable object (e.g., the thickness of the region 7080-d remains unchanged in fig. 7Q-7W because the input from the user 7002 does not point to the region 7080-d and does not indicate that the user 7002 is ready to interact with the region 7080-d in fig. 7R-7V, and because no other input is detected that indicates that the user 7002 is ready to interact with the region 7080-d). Changing the thickness of the selectable object in response to an input indicating that the user is ready (intended) to interact with the selectable object indicates which portion of the user interface currently has focus for further interaction, which provides improved visual feedback regarding the state of the computer system.

In some implementations, determining that the first input indicates that the user is ready to interact with the first selectable object includes determining that the user is focusing on the first selectable object (e.g., a gaze of the user is directed at the first selectable object). For example, as shown in fig. 7R-7T and described with reference to these figures, it is determined that the user 7002 is ready to interact with a respective selectable region of the user interface element 7080 based on the user 7002 focusing on the respective selectable region. Requiring the user to focus on the selectable object to enable interaction with the selectable object allows the computer system to more accurately automatically discern the user's intent and thereby reduce input errors and unintended results.

In some implementations, determining that the first input indicates that the user is ready to interact with the first selectable object includes determining that the user's hand is in a ready state (e.g., and optionally, determining that the user is looking at the first selectable object, plus the user's hand is in a ready state). For example, as shown in and described with reference to fig. 7R, the user 7002 is determined to be ready to interact with a respective selectable region of the user interface element 7080 based on the user's 7002 hand being in a ready state (as an alternative or supplement to the user 7002 focusing on the respective selectable region). The user is required to put their hand in a ready state to enable interaction with the selectable object so that the computer system more accurately automatically discerns the user's intent and thereby reduces input errors and unintended results.

In some embodiments, changing the thickness of the first selectable object includes displaying an animation of the thickness of the first selectable object gradually increasing (e.g., increasing from the first thickness to the second thickness). For example, as described with reference to fig. 7R, the thickness increase of the corresponding region, such as the region 7080-b (or the similar region 7080-c (fig. 7S)), is displayed via animation of the thickness increase. Animating the increase in thickness of the selectable object in response to detecting the user's intent to interact with the selectable object provides improved visual feedback regarding the state of the computer system in a less visually discordant manner.

In some embodiments, the computer system indicates the thickness of the first selectable object by displaying a visual effect of reflecting simulated light from one or more edges of the first selectable object (e.g., based on the respective simulated material of the first selectable object). For example, as shown in fig. 7R-7V and described with reference to these figures, when the respective selectable regions of the user interface element 7080 have an increased thickness, specular reflection from one or more edges of the respective selectable regions is displayed and indicative of the thickness. In some embodiments, the degree of visual effect of the reflected light is based on the thickness of the first selectable object. In some embodiments, the visual effect (e.g., the degree of visual effect) of light reflected from the one or more edges of the first selectable object is gradually updated as the thickness of the first selectable object gradually changes (e.g., increases as the animation proceeds). Displaying simulated specular reflection from edges of the selectable object to indicate the thickness of the object improves the user's situational awareness and provides improved visual feedback regarding the state of the computer system.

In some embodiments, in conjunction with changing (e.g., increasing) the thickness of the first selectable object, the computer system changes (e.g., increases) the amount by which the visual effect of the reflected simulated light from the one or more edges of the first selectable object extends toward the center of the first selectable object (e.g., for increasing the thickness of the first selectable object, including increasing the apparent amount of light reflected from the first selectable object, such as by showing that the reflection extends further along the one or more edges of the first selectable object and/or that the display light is reflected further internally within the first selectable object if the first selectable object simulates a transparent or translucent simulated material). For example, as described with reference to fig. 7R, as the thickness of the respective region changes, the amount and/or extent of specular reflection from the respective region, such as region 7080-b (or similar region 7080-c (fig. 7S)), correspondingly changes. Changing the amount and degree of simulated specular reflection from the edge of the selectable object based on the change in thickness of the object improves the user's situational awareness and provides improved visual feedback regarding the state of the computer system.

In some embodiments, the first view of the three-dimensional environment is visible from the viewpoint of the user, and in response to detecting the first input, and in accordance with a determination that the first input indicates that the user is ready to interact with the first selectable object (e.g., in connection with changing a thickness of the first selectable object), the computer system reduces a simulated distance between the first selectable object and the viewpoint of the user (e.g., displays the first selectable object as appearing to move toward the viewpoint of the user). For example, as shown in fig. 7R and 7T and described with reference to these figures, in response to the user 7002 focusing on the region 7080-b, the region 7080-b is separated from and lifted off the surface of the user interface element 7080 and brought closer to the viewpoint of the user 7002. In another example, as shown in fig. 7S and described with reference thereto, in response to the user 7002 focusing on the region 7080-c, the region 7080-c is separated from and lifted off the surface of the user interface element 7080 and brought closer to the viewpoint of the user 7002. In response to an input indicating that the user is ready to interact with the selectable object, the selectable object is moved closer to the user's point of view emphasizing that the selectable object has focus for further interaction, which provides improved visual feedback regarding the state of the computer system.

In some embodiments, reducing the simulated distance between the first selectable object and the user's point of view increases the degree of separation between the first selectable object and the surface (e.g., back surface) of the user interface object (e.g., displaying the first selectable object as appearing separate from the user interface object and moving away from the user interface object (e.g., surface of the user interface object, such as back surface) toward the user's point of view, without generally changing the simulated distance between the user's point of view and the user interface object). In some embodiments, increasing the degree of separation between the first selectable object and the surface of the user interface object changes (e.g., increases) the z-height of the first selectable object relative to the user interface object. For example, as shown in fig. 7R and 7T and described with reference to these figures, in response to the user 7002 focusing on the region 7080-b, the region 7080-b is separated from the surface of the user interface element 7080 and lifted off the surface (e.g., from the front surface and by extending from the back surface). In another example, as shown in fig. 7S and described with reference thereto, in response to the user 7002 focusing on the region 7080-c, the region 7080-c is separated from the surface of the user interface element 7080 and lifted off the surface (e.g., from the front surface and by extending from the back surface). As used herein, the thickness of a respective object refers to the dimension of the object (e.g., a measurement of the object from its front surface to its back surface). As used herein, the height (also referred to as z-height) of a respective object refers to the degree of separation between the object and another object (e.g., rather than the size of the object). Lifting a selectable object that is part of a user interface object away from other parts (e.g., the rest) of the user interface object emphasizes that the selectable object has focus for further interaction while weakening other aspects of the user interface object from having focus, which provides improved visual feedback regarding the state of the computer system.

In some embodiments, the computer system displays simulated shadows cast by the first selectable object onto the surface of the user interface object (e.g., based on one or more simulated light sources in a three-dimensional environment and/or one or more real light sources in a physical environment surrounding the display generating component). In some implementations, the shadow is displayed in conjunction with increasing the degree of separation between the first selectable object and the surface of the user interface object (e.g., if the surface of the first selectable object is initially flush with the surface of the user interface object, then the shadow is not initially displayed). For example, as shown in fig. 7R-7T and described with reference to these figures, shadows are displayed as cast onto the user interface element 7080 when the respective selectable regions of the user interface element 7080 have an increased thickness. Displaying simulated shadows cast by selectable objects onto an associated user interface object (e.g., a background tray) further emphasizes the selectable object with focus relative to other portions of the user interface object for further interaction, which provides improved visual feedback regarding the state of the computer system.

In some embodiments, the user interface object further comprises a second selectable object that is capable of being activated to perform a second operation in the three-dimensional environment, wherein the second selectable object is different from the first selectable object and the second operation is different from the respective operation. In some embodiments, reducing the simulated distance between the first selectable object and the viewpoint of the user moves the first selectable object relative to the second selectable object (e.g., changes the degree of separation or z-height of the first selectable object relative to the second selectable object). In some embodiments, in response to an input indicating that the user is ready to interact with the first selectable object (e.g., rather than with the second selectable object), the second selectable object does not move relative to the user interface object and/or does not move relative to the viewpoint of the user when the first selectable object moves away from the user interface object. For example, as shown in FIG. 7R and described herein with reference to that figure, in response to the user 7002 focusing on the region 7080-b, the region 7080-b is brought closer to the viewpoint of the user 7002, thereby moving the region 7080-b relative to the non-moving regions 7080-c and 7080-d. The selectable object that is promoted as part of the user interface object away from the other selectable object that is part of the user interface object emphasizes the selectable object with focus for further interaction while weakening the other selectable portion of the user interface object from having focus, which provides improved visual feedback regarding the state of the computer system.

In some embodiments, in response to detecting the first input, and in accordance with a determination that the first input indicates that the user is ready to interact with the first selectable object (e.g., in connection with changing a thickness of the first selectable object), the computer system changes (e.g., reduces) an opacity of the first selectable object. For example, as shown in fig. 7R and 7T and described with reference to these figures, in response to the user 7002 focusing on the region 7080-b, the opacity of the region 7080-b is reduced relative to the opacity of the region in fig. 7Q. In another example, as shown in FIG. 7S and described with reference thereto, in response to the user 7002 focusing on the region 7080-c, the opacity of the region is reduced relative to the opacity of the region 7080-c in FIG. 7Q. In some embodiments, the opacity of the first selectable object is increased. In some embodiments, in response to detecting the first input, and in accordance with a determination that the first input indicates that the user is ready to interact with the first selectable object, the computer system changes one or more other visual properties of the first selectable object (e.g., increases or decreases brightness, increases or decreases saturation, and/or changes hue). For example, the brightness of the first selectable object is increased. In response to detecting a user's intent to interact with the selectable object, the opacity of the selectable object is changed, visually distinguishing the selectable object with focus for further interaction, which provides improved visual feedback regarding the state of the computer system.

In some embodiments, in response to detecting the first input, and in accordance with a determination that the first input indicates that the user is ready to interact with the first selectable object (e.g., in connection with changing a thickness of the first selectable object), the computer system changes (e.g., increases) a degree of blur of the first selectable object (e.g., changes, such as increases, a blur radius of the first selectable object). For example, as described herein with reference to fig. 7R and 7T, in response to the user 7002 focusing on the region 7080-b, the ambiguity of the content below the region 7080-b (as observed by the region 7080-b in fig. 7R and 7T) increases relative to the ambiguity of the content below the region 7080-b (as observed by the region 7080-b in fig. 7Q). In another example, as shown in fig. 7S and described herein with reference to the figure, in response to the user 7002 focusing on the region 7080-c, the ambiguity of the content below the region 7080-c (as observed by the region 7080-c in fig. 7S) increases relative to the ambiguity of the content below the region 7080-c (as observed by the region 7080-c in fig. 7Q). In some embodiments, the blurring of the first selectable object is applied to the appearance of the first selectable object within the existing boundary of the first selectable object (e.g., the blurring does not extend the boundary of the first selectable object (e.g., does not eclipse or soften the edge)). In some embodiments, where the appearance of the first selectable object is based on the appearance of a portion of the three-dimensional environment behind the first selectable object (e.g., farther from the user's viewpoint than the simulated position of the first selectable object in the three-dimensional environment), blurring of the first selectable object is applied to the appearance of an underlying portion of the three-dimensional environment (e.g., how the underlying portion of the three-dimensional environment is "rendered through" the first selectable object is changed by the degree of blurring of the first selectable object), for example, in order to simulate the appearance of a transparent or translucent material. Changing the degree of blurring of the selectable object in response to detecting the user's intent to interact with the selectable object provides improved visual feedback regarding the state of the computer system, in some cases including enhancing the visual effect of the simulated translucent material lifting off of the underlying object.

In some embodiments, the user interface object further comprises a second selectable object that is capable of being activated to perform a second operation in the three-dimensional environment, wherein the second selectable object is different from the first selectable object and the second operation is different from the respective operation. In some embodiments, changing the thickness of the first selectable object (e.g., in response to the first input if the first input indicates that the user is ready to interact with the first selectable object) changes the thickness of the first selectable object relative to the thickness of the second selectable object. For example, as shown in FIG. 7R and described herein with reference to this figure, in response to the user 7002 focusing on the region 7080-b, the thickness of the region 7080-b increases, thereby changing the thickness of the region 7080-b relative to the thicknesses of the regions 7080-c and 7080-d, thereby being different from the thicknesses of the regions 7080-c and 7080-d, which remain unchanged from FIG. 7Q. In some embodiments, the thickness of the second selectable object does not change in response to an input indicating that the user is ready to interact with the first selectable object (e.g., rather than interacting with the second selectable object). The thickness of the selectable object that changes to be part of the user interface object relative to the thickness of other selectable objects that are part of the user interface object emphasizes the selectable object with focus for further interaction while weakening other selectable parts of the user interface object to be out of focus, which provides improved visual feedback regarding the state of the computer system.

In some embodiments, the user interface object further comprises a second selectable object that is capable of being activated to perform a second operation in the three-dimensional environment, wherein the second selectable object is different from the first selectable object and the second operation is different from the respective operation. In some embodiments, the computer system: detecting a second input corresponding to a second selectable object, the second input indicating that the user is ready to interact with the second selectable object; and in response to detecting the second input (e.g., and in accordance with a determination that the second input indicates that the user is ready to interact with the second selectable object), changing (e.g., increasing) a thickness of the second selectable object (e.g., from a third thickness, optionally the same as the first thickness, to a fourth thickness, optionally the same as the second thickness). In some embodiments, the second selectable object occupies a respective region of the user interface object, and changing the thickness of the second selectable object includes changing a degree of separation between a surface of the second selectable object and a surface of a region of the user interface object adjacent to the respective region. As described herein with respect to the first selectable object, in some embodiments, the degree of separation varies from zero (e.g., from no separation to some separation). For example, as shown in FIG. 7S and described herein with reference to that figure, the thickness of the region 7080-c increases in response to the user 7002 focusing on the region 7080-c. Changing the thickness of a particular selectable object in response to an input indicating that the user is ready to interact with the particular selectable object indicates which portion of the user interface currently has focus for further interaction, which provides improved visual feedback regarding the state of the computer system.

In some embodiments, changing the thickness of the first selectable object changes the thickness of the first selectable object by a corresponding amount, and changing the thickness of the second selectable object changes the thickness of the second selectable object by the same corresponding amount. For example, in FIG. 7R, the thickness increase of the region 7080-b in response to the user 7002 being focused on the region 7080-b is the same as the thickness increase of the region 7080-c in FIG. 7S in response to the user 7002 being focused on the region 7080-c. Changing the thickness of different selectable objects by the same amount provides consistent visual feedback regarding the same type of state change at the computer system.

In some implementations, detecting the second input includes detecting that the user's attention has been diverted from the first selectable object to the second selectable object (e.g., detecting the second input concurrently with or after ceasing to detect the first input indicating that the user is ready to interact with the first selectable object). In some implementations, detecting the second input includes detecting that the user's attention has moved from the first selectable object to the second selectable object while the user's hand remains in a ready state (e.g., as described herein, the user's hand being in a ready state is one of the requirements for ready interaction with the selectable object and is thus a precondition for various types of visual feedback provided in response to detecting ready interaction). In some embodiments, in response to detecting the second input (e.g., in conjunction with changing the thickness of the second selectable object), the computer system reverses the change in thickness of the first selectable object (e.g., a change performed in response to the first input indicating that the user is ready to interact with the first selectable object). For example, in response to the user 7002 being distracted from the region 7080-b (fig. 7R) to the region 7080-c (fig. 7S), the thickness of the region 7080-b (fig. 7S) reverts from its appearance in fig. 7R to its appearance in fig. 7Q. In another example, in response to the user 7002' S attention being diverted from region 7080-c (fig. 7S) to region 7080-b (fig. 7T), the thickness of region 7080-c (fig. 7T) reverts from its appearance in fig. 7S to its appearance in fig. 7Q. Conversely, when the user switches from indicating that interaction with the first selectable object is ready to indicate that interaction with the second selectable object is ready, a previous change in the thickness of the second selectable object is changed and the thickness of the first selectable object is reversed (e.g., only the changed thickness of the selectable object with which the user is currently indicating that interaction is ready is displayed) indicates which portion of the user interface currently has focus for further interaction, which provides improved visual feedback regarding the state of the computer system.

In some implementations, after detecting the first input, and in accordance with a determination that the first input indicates that the user is ready to interact with the first selectable object, the computer system detects a first portion of the interactive input directed to the first selectable object (e.g., a first portion of the input gesture performed by the hand 7020 of the user 7002 in fig. 7U). In some embodiments, the interaction input comprises a gesture (e.g., an air gesture such as an air pinch or an air tap) and optionally must be initiated when the user is indicating that it is ready to interact with the first selectable object (e.g., initiated when the user focuses on the first selectable object, such as by looking at the first selectable object, and/or performed by the user's hand from a ready state). In some embodiments, in response to detecting a first portion of the interactive input directed to the first selectable object (e.g., before detecting an end of the interactive input), the computer system changes an appearance of the first selectable object. One of ordinary skill in the art will recognize that changing the respective visual attribute (e.g., size, opacity, brightness, blur amount, or other visual attribute described herein) of the first selectable object includes increasing the value, amount, or degree of the respective visual attribute, or alternatively decreasing the value, amount, or degree of the respective visual attribute, as described herein. For example, as described herein with reference to fig. 7U, in response to an input gesture performed by the hand 7020 of the user 7002 (e.g., a first portion of the input gesture or initiation of the input gesture), the appearance of the region 7080-b is changed when the region 7080-b is selected for further interaction. Further changing the appearance of the selectable object in response to interaction with the selectable object (e.g., the beginning of the interaction) after the thickness of the selectable object has been changed in response to the user merely indicating that interaction with the selectable object is ready, visually emphasizes the portion of the user interface that is currently receiving user input, which provides improved visual feedback regarding the state of the computer system.

In some embodiments, changing the appearance of the first selectable object in response to detecting the first portion of the interactive input includes changing (e.g., decreasing, or in some embodiments increasing) the size (e.g., scaling down) of the first selectable object. For example, in fig. 7U, the size of the lift region 7080-b decreases in response to an input gesture performed by the hand 7020. Changing the size of the selectable object in response to interaction with the selectable object visually distinguishes portions of the user interface that are currently receiving user input, which provides improved visual feedback regarding the state of the computer system.

In some embodiments, changing the appearance of the first selectable object in response to detecting the first portion of the interactive input includes changing (e.g., increasing, or in some embodiments decreasing) the opacity of the first selectable object. For example, in fig. 7U, the opacity of the lift region 7080-b is reduced (e.g., further) in response to an input gesture performed by the hand 7020. Changing the opacity of the selectable object in response to interaction with the selectable object improves legibility of the selectable object during interaction and visually distinguishes portions of the user interface currently receiving user input, which provides improved visual feedback regarding the state of the computer system.

In some embodiments, changing the appearance of the first selectable object in response to detecting the first portion of the interactive input includes changing (e.g., decreasing, or in some embodiments increasing) the brightness of the first selectable object. For example, in fig. 7U, the brightness of the lift region 7080-b decreases in response to an input gesture performed by the hand 7020. Changing the brightness of the selectable object in response to interaction with the selectable object improves legibility of the selectable object during interaction and visually distinguishes portions of the user interface currently receiving user input, which provides improved visual feedback regarding the state of the computer system.

In some implementations, changing the appearance of the first selectable object in response to detecting the first portion of the interaction input includes changing (e.g., increasing) an intensity (e.g., a length, darkness, and/or sharpness or ambiguity) of a simulated shadow of the first selectable object (e.g., displayed on or appearing to be projected by the first selectable object) (e.g., as projected by the first selectable object, the user interface object, and/or other objects in the three-dimensional environment based on real (from a corresponding physical environment) and/or simulated light in the three-dimensional environment). For example, in fig. 7U, it appears that the shadow cast by the lift region 7080-b is enhanced in response to an input gesture performed by the hand 7020. Changing the intensity of the simulated shadow of the selectable object in response to interaction with the selectable object visually distinguishes the portion of the user interface that is currently receiving user input, which provides improved visual feedback regarding the state of the computer system.

In some embodiments, changing the appearance of the first selectable object in response to detecting the first portion of the interaction input includes changing the appearance of the first selectable object relative to the first set of one or more visual attributes without changing (e.g., while maintaining) the appearance of the first selectable object relative to the second set of one or more visual attributes, wherein the first set of one or more visual attributes is different from the second set of one or more visual attributes (e.g., the two sets include different visual attributes, and in some embodiments do not include a common visual attribute). For example, as shown in fig. 7U and described herein with reference to that figure, while some visual properties (such as size, opacity, and brightness) of the region 7080-b are changed in response to an input gesture performed by the hand 7020, other visual properties (such as thickness, specular reflection, and separation from the surface of the user interface element 7080) remain unchanged. In some examples, in response to the interaction input, the appearance of the first selectable object changes relative to size, opacity, brightness, and/or shading intensity (e.g., in the first set of visual properties), and the appearance of the first selectable object does not change relative to thickness, z-height (e.g., relative to the user interface object), blur, and/or specular reflection (e.g., in the second set of visual properties). In some implementations, the appearance of the first selectable object does not change relative to the relative spatial position of the first selectable object and the user interface object (e.g., the centroid of the first selectable object does not move in the x or y direction relative to the centroid of the user interface object, optionally while also maintaining the relative z position of the centroid of the first selectable object and the centroid of the user interface object). In some embodiments, similarly, a change in one or more visual properties of the first selectable object is performed without changing one or more other visual properties (e.g., thickness, specular reflection, z-height, shading, opacity, and/or blur without changing scale) in response to detecting the first input and in accordance with a determination that the first input indicates that the user is ready to interact with the first selectable object. Maintaining one or more other visual properties of the selectable object while changing one or more visual properties of the selectable object in response to interaction with the selectable object balances providing improved visual feedback regarding the state of the computer system with excessive changes that clutter the user interface that would otherwise increase the likelihood that the user would suffer from motion sickness when using the computer system.

In some implementations, in response to detecting the first portion of the interactive input, the computer system maintains a thickness (e.g., at the second thickness) of the first selectable object. For example, in fig. 7U, the thickness of the lift region 7080-b is maintained at the same thickness as in fig. 7T in response to an input gesture performed by the hand 7020. Maintaining the thickness of the selectable object while changing one or more other visual properties of the selectable object in response to interaction with the selectable object balances providing improved visual feedback regarding the state of the computer system with excessive changes that clutter the user interface that would otherwise increase the likelihood that the user would suffer motion sickness when using the computer system.

In some implementations, in response to detecting the first portion of the interaction input, the computer system maintains a degree of separation between the first selectable object and a surface of the user interface object (e.g., and maintains a simulated distance between the first selectable object and a viewpoint of the user). For example, the amount by which the region 7080-b is lifted off the surface of the user interface element 7080 is the same in FIG. 7U as in FIG. 7T. Maintaining the z-height of the selectable object relative to another associated user interface object while changing one or more other visual properties of the selectable object in response to interaction with the selectable object balances providing improved visual feedback regarding the state of the computer system with excessive changes that clutter the user interface that would otherwise increase the likelihood that the user would suffer from motion sickness when using the computer system.

In some embodiments, in response to detecting the first portion of the interactive input, the computer system maintains a degree of blur for the first selectable object. For example, the amount of blurring of the underlying content viewed through the region 7080-b is the same in FIG. 7U as in FIG. 7T (e.g., the thickness of the region 7080-b and the degree of separation from the user interface element 7080 is the same in FIG. 7U as in FIG. 7T). Maintaining the degree of blurring of the selectable object while changing one or more other visual properties of the selectable object in response to interaction with the selectable object balances providing improved visual feedback regarding the state of the computer system with excessive changes that clutter the user interface that would otherwise increase the likelihood that the user would suffer motion sickness when using the computer system.

In some implementations, in response to detecting the first portion of the interactive input, the computer system maintains a visual effect of (e.g., a degree of) light being reflected from one or more edges of the first selectable object (e.g., according to maintaining a thickness of the first selectable object). For example, the specular reflection from the raised region 7080-b is the same in FIG. 7U as in FIG. 7T (e.g., the thickness of the region 7080-b is the same in FIG. 7U as in FIG. 7T). Maintaining the appearance of simulated specular reflection from the selectable object while changing one or more other visual properties of the selectable object in response to interaction with the selectable object balances providing improved visual feedback regarding the state of the computer system with excessive changes that clutter the user interface that would otherwise increase the likelihood that the user would suffer from motion sickness when using the computer system.

In some implementations, the computer system detects a second portion of the interactive input that includes an end of the interactive input (e.g., the second portion immediately follows the first portion). In some implementations, a first portion of the interactive input corresponds to a first portion of the gesture (e.g., a first portion of an air pinch or a tap that brings two fingers into contact with each other), and a second portion of the interactive input corresponds to a second portion of the gesture (e.g., release of the air pinch or release of the tap by moving two fingers away from each other after bringing the two fingers into contact with each other). In some embodiments, in response to detecting the second portion of the interaction input, the computer system changes the appearance of the first selectable object (e.g., in a different manner than the second manner of changing the appearance of the first selectable object in response to detecting the first portion of the interaction input and/or in a different third manner than the first manner of changing the appearance of the first selectable object in response to detecting the first input indicating that the user is ready to interact with the first selectable object) and performs a corresponding operation in the three-dimensional environment (e.g., activates a button or other control, launches an application from an application icon, displays a menu, changes the state of a toggle switch, or focuses on an input field such as a text box). In some embodiments, the appearance of the first selectable object is changed in response to detecting a portion of the second portion of the interactive input occurring before the interactive input ends. In some implementations, changing one or more visual properties (e.g., size, opacity, brightness, shading intensity, and/or thickness or etch depth) of the first selectable object in response to detecting the second portion of the interactive input at least partially inverts a previous change in the one or more visual properties of the first selectable object (e.g., a previous change performed in response to detecting the first portion of the interactive input). For example, in response to the end of the input gesture performed by the hand 7020 in fig. 7U, the appearance of the region 7080-b changes from the appearance shown in fig. 7U to the appearance shown in fig. 7V (e.g., the region 7080-b increases in size, opacity, and brightness) and a corresponding operation associated with the region 7080-b is performed. In another example, in response to the end of the input gesture performed by the hand 7020 in fig. 7U and in accordance with a determination that the attention of the user 7002 is no longer focused on the region 7080-b as in fig. 7W, a corresponding operation associated with the region 7080-b is performed, and thus the appearance of the region 7080-b changes from the appearance shown in fig. 7U to the appearance shown in fig. 7W (e.g., the thickness of the region 7080-b and the degree of separation from the user interface element 7080 also decrease). Changing the appearance of the selectable object and performing associated operations in response to completion of the user interaction with the selectable object, particularly when the appearance of the selectable object is changed in a different manner than an earlier portion of the user interaction, provides improved visual feedback regarding the state of the computer system.

In some implementations, changing the appearance of the first selectable object in response to detecting the second portion of the interactive input includes changing (e.g., increasing) a size (e.g., scaling up) of the first selectable object. For example, in response to the end of an input gesture performed by the hand 7020 (shown in fig. 7U), the size of the region 7080-b in fig. 7V-7W increases (e.g., reverses) relative to the size of the region 7080-b in fig. 7U. Changing the size of the selectable object in response to completion of the user interaction with the selectable object provides improved visual feedback regarding the state of the computer system.

In some embodiments, changing the appearance of the first selectable object in response to detecting the second portion of the interactive input includes changing (e.g., decreasing, or in some embodiments increasing) the opacity of the first selectable object. For example, in response to the end of an input gesture performed by the hand 7020 (shown in fig. 7U), the opacity of the region 7080-b in fig. 7V-7W increases relative to the opacity of the region 7080-b in fig. 7U. Changing the opacity of the selectable object in response to completion of the user interaction with the selectable object provides improved visual feedback regarding the state of the computer system.

In some implementations, changing the appearance of the first selectable object in response to detecting the second portion of the interactive input includes changing (e.g., increasing or decreasing) the brightness of the first selectable object. For example, in response to the end of an input gesture performed by the hand 7020 (shown in fig. 7U), the brightness of the region 7080-b in fig. 7V-7W is different from the brightness of the region 7080-b in fig. 7U (e.g., the brightness increases relative to the region 7080-b in fig. 7U). Changing the brightness of the selectable object in response to completion of the user's interaction with the selectable object provides improved visual feedback regarding the state of the computer system.

In some embodiments, changing the appearance of the first selectable object in response to detecting the second portion of the interactive input includes changing (e.g., decreasing or increasing) the intensity of a simulated shadow of the first selectable object (e.g., displayed on or appearing to be projected by the first selectable object). For example, shadows that appear to be cast by the region 7080-b onto the user interface element 7080 are less intense in fig. 7V than in fig. 7U. Changing the intensity of the simulated shadow of the selectable object in response to completion of the user interaction with the selectable object provides improved visual feedback regarding the state of the computer system.

In some implementations, after detecting the end of the interactive input (e.g., and/or after performing a corresponding operation), the computer system maintains a thickness (e.g., a second thickness) of the first selectable object while the user's attention remains focused on the first selectable object. In some embodiments, in accordance with a determination that the user's attention remains focused on the first selectable object when the end of the interactive input is detected, the thickness of the first selectable object is maintained until the user's attention ceases to be focused on the first selectable object. For example, after the input gesture performed by the hand 7020 (shown in FIG. 7U) ends, the increased thickness of the region 7080-b in FIG. 7U is maintained in FIG. 7V, while the attention of the user 7002 remains focused on the region 7080-b in FIG. 7V. Maintaining the thickness of the selectable object while the user's attention remains focused on the selectable object indicates that the selectable object has focus even after user interaction with the selectable object has ended, which provides visual feedback regarding the state of the computer system.

In some embodiments, changing the appearance of the first selectable object in response to detecting the second portion of the interactive input includes changing (e.g., decreasing or increasing) a thickness of the first selectable object (e.g., returning from the second thickness to the first thickness, or changing to a third thickness different from the first thickness). In some embodiments, the thickness of the first selectable object is changed after the respective operation is performed. In some embodiments, the thickness of the first selectable object is changed after the second portion of the interaction input is detected and/or after the corresponding operation is performed, regardless of whether the user's attention remains focused on the first selectable object, and optionally even when the user's attention remains focused on the first selectable object (e.g., even when the user continues to indicate that it is ready to interact with the first selectable object). For example, the thickness of the region 7080-b in fig. 7W decreases in response to the end of an input gesture performed by the hand 7020 (shown in fig. 7U), and such thickness reduction optionally occurs regardless of whether the user 7002 remains focused on the region 7080-b, as described herein with reference to fig. 7W. Even if the user's attention remains focused on the selectable object, changing the thickness of the selectable object in response to completion of the user's interaction with the selectable object provides improved visual feedback regarding the state of the computer system.

In some embodiments, a first input corresponding to the first selectable object is detected while the first selectable object is displayed as having a first appearance. In some implementations, after detecting the end of the interactive input (e.g., after changing the appearance of the first selectable object in response to detecting the second portion of the interactive input), the computer system displays the first selectable object with a second appearance that is different from the first appearance. In some embodiments, if the first selectable object is a button, the button is displayed before being activated as having a different appearance than after being activated (e.g., optionally in combination with a previous first input indicating that interaction is ready in the event that activation occurs in response to the first and/or second portions of interaction input). For example, for a toggle switch, the toggle switch is displayed with a first appearance corresponding to a first state and when activated is displayed with a second appearance corresponding to a second state (e.g., from an "on" state to an "off" state when activated, and vice versa). For example, before interacting with the region 7080-b (e.g., activating the region) via the input gesture of fig. 7U, the region 7080-b in fig. 7W has a different appearance than the region 7080-b in fig. 7Q and 7S after interacting with the region 7080-b (e.g., activating the region) via the input gesture of fig. 7U. Changing the appearance of the selectable object having the first idle state appearance before the user indicates that the interaction is ready and then interacts with the selectable object to the second idle state appearance after the user completes the interaction with the selectable object indicates that the selectable object has been activated provides improved visual feedback regarding the state of the computer system.

In some embodiments, in accordance with a determination that the first selectable object is an interactive object of a first type (e.g., a button, toggle, or slider), changing the thickness of the first selectable object (e.g., in response to detecting a user input indicating that the user is ready to interact with the first selectable object) includes moving the first selectable object away from the user interface object (e.g., lifting and/or separating in a direction perpendicular to a surface of the user interface object). For example, in response to the user 7002 focusing on the region 7080-b (fig. 7R and 7T), the region 7080-b is moved away from the user interface element 7080. In another example, in response to the user 7002 focusing on the region 7080-c (fig. 7S), the region 7080-c is moved away from the user interface element 7080. In some embodiments, in accordance with a determination that the first selectable object is an interactive object of a second type (e.g., such as an input field for text input), changing the thickness of the first selectable object includes increasing a depth of the first selectable object into the user interface object (e.g., in a direction perpendicular to a surface of the user interface object). For example, in fig. 7O, in response to the user 7002 focusing on the recessed region 7054-b (e.g., the recessed indication region 7054-b of the region 7054-b is a different type of interactive object than the regions 7080-b, 7080-c, and 7080-d (fig. 7O)), the recessed region 7054-b is further recessed (e.g., increased in depth) from the surface of the user interface element 7054. In some implementations, recessing the first selectable object into the user interface object moves a surface (e.g., a front surface) of the first selectable object from being flush with the surface (e.g., the front surface) of the user interface object to being farther from a viewpoint of the user (e.g., in a direction perpendicular to the surface of the user interface object) than the surface of the user interface object. Changing the thickness of certain types of selectable objects by lifting them off the associated user interface object and changing the thickness of other types of selectable objects by recessing them further into the associated user interface object indicates what types of objects the selectable objects are, and thus what types of interactions the selectable objects support, which provides improved visual feedback regarding the state of the computer system.

In some embodiments, changing the thickness of the first selectable object (e.g., if the first selectable object is an input field such as a text box or a search bar) includes increasing the depth of the first selectable object recess (e.g., extending) into the front surface (e.g., the surface facing the user viewpoint) of the user interface object. For example, in fig. 7O, in response to the user 7002 focusing on the recessed region 7054-b, the recessed region 7054-b is further recessed (e.g., increased in depth) from the surface of the user interface element 7054. In some embodiments, the first selectable object is an area recessed (e.g., etched) from a surface of the user interface object (e.g., and the first selectable object is displayed with one or more specular reflections and/or with one or more shadows cast by, for example, the user interface object), and changing the thickness of the first selectable object (e.g., in response to detecting the first input, and in accordance with a determination that the first input indicates that the user is ready to interact with the first selectable object) includes further recessing the first selectable object into the surface of the user interface object (e.g., further etching the surface) (e.g., and optionally increasing the degree and/or intensity of specular reflection from and/or shadows cast onto the first selectable object, respectively). In some embodiments, alternatively or additionally, in response to detecting the interactive input (e.g., a first portion of the interactive input) described herein, further recessing the first selectable object into the user interface object is performed. Further recessing the selectable object that is part of the associated user interface object into the user interface object in response to an input indicating that the user is ready to interact with the selectable object indicates which portion of the user interface currently has focus for further interaction, what type of object the selectable object is, and thus what type of interaction the selectable object supports, provides improved visual feedback regarding the state of the computer system.

In some embodiments, based on the change in thickness of the first selectable object, the depth of the first selectable object recess into the front surface of the user interface object is increased without changing the back surface of the user interface object (e.g., the surface opposite the front surface, the surface facing away from the user's viewpoint). In some embodiments, the back surface of the user interface object remains unchanged because no protrusions from the back surface of the user interface object or no further protrusions are generated from the first selectable object to the depression in the front surface of the user interface object. For example, in fig. 7O, side view 7055 indicates that as recessed region 7054-b is further recessed from the surface of user interface element 7054, the back surface of user interface element 7054 is unchanged. Responsive to an input indicating that a user is ready to interact with a selectable object, the selectable object, which is part of the front surface of the associated user interface object, is further recessed into the front surface of the user interface object without changing the back surface of the user interface object, which indicates the type of input being detected (e.g., intent to interact with the selectable object rather than a request for lateral movement of the selectable object), which provides improved visual feedback regarding the state of the computer system.

In some embodiments, the computer system displays text in the first selectable object and detects the first input when the text is displayed at a first simulated distance from the user's point of view (e.g., a first depth relative to the recessed first selectable object, such as at a forwardmost plane of the user interface object into which the first selectable object is recessed (e.g., depth is zero)). In some embodiments, in response to detecting the first input, and in accordance with a determination that the first input indicates that the user is ready to interact with the first selectable object, the computer system displays text in the first selectable object at a second (e.g., greater) simulated distance relative to the user's point of view (e.g., moves text to a different simulated distance from the user's point of view, and optionally to a different second depth relative to the recessed first selectable object, such as beyond a frontmost plane (e.g., non-zero depth) of the user interface object). In some implementations, the amount by which the text is moved is based on (e.g., equal to, proportional to) an amount by which the thickness of the first selectable object changes in response to the first input (e.g., maintaining a relative depth of the text relative to the first selectable object). In some implementations, in response to detecting the interactive input (e.g., a first portion of the interactive input), displaying text at a different (e.g., increased) simulated distance relative to the user's point of view is performed. In some implementations, the second simulated distance is the same as the first simulated distance (e.g., text does not move relative to the user's viewpoint and/or relative to the user interface object when the thickness (e.g., etch depth) of the first selectable object changes). Changing the depth of text in the recessed area in response to an input indicating that the user is ready to interact with the recessed area is described in more detail herein with reference to fig. 7O. Changing the depth of text in the recessed selectable object in response to an input indicating that the user is ready to interact with the selectable object provides improved visual feedback regarding the state of the computer system.

In some embodiments, after detecting the first input, and in accordance with a determination that the first input indicates that the user is ready to interact with the first selectable object, the computer system detects an interaction input directed to the first selectable object (e.g., such as by selecting a text field to cause the first selectable object to be activated by the inputtable text), wherein the interaction input includes a first portion followed by a second portion that includes an end of the interaction input. In some implementations, in response to detecting the first portion of the interactive input, the computer system changes (e.g., increases) a distance between text in the first selectable object and a viewpoint of the user. In some implementations, in response to detecting the second portion (e.g., end) of the interactive input, the computer system reverses a change in distance between text in the first selectable object and the user's viewpoint (e.g., this is performed in response to detecting the first portion of the interactive input). In some implementations, any change in distance between text displayed in response to detecting the first input and the viewpoint of the user is reversed. Changing the depth of text in the recessed area in response to an input gesture interacting with the recessed area (e.g., wherein the input gesture is initiated when the user is indicating that it is ready to interact with the recessed area) is described in more detail herein with reference to fig. 7O. Changing the depth of text in the recessed selectable object in response to the beginning and subsequent ending of the user interaction with the selectable object, and then reversing the change in depth, which indicates the progress of the user interaction, provides improved visual feedback regarding the state of the computer system, particularly if no other visual feedback such as a change in depth, brightness, and/or size is provided for the selectable object in response to the user interaction (although other visual feedback is provided for non-recessed selectable objects).

In some implementations, determining that the first input indicates that the user is ready to interact with the first selectable object includes determining that the user's attention (e.g., gaze) is focused on the first selectable object and that the user's hand is in a ready state. In some embodiments, the first selectable object includes a slider. In some embodiments, in response to detecting a first input corresponding to a first selectable object: when the first input indicates that the user is ready to interact with the first selectable object, the computer system displays an adjustment control for the slider (e.g., a slider thumb or other control for changing the current value selected by the slider, and/or other indication of the current value selected by the slider); and when the first input does not indicate that the user is ready to interact with the first selectable object (e.g., the user's gaze is not focused on the first selectable object and/or the user's hand is not in a ready state), the computer system displays the slider without displaying an adjustment control for the slider. The display and behavior (e.g., change in appearance) of a slider control for changing the current value of a slider is described in more detail herein with reference to fig. 7R and 7U-7V. Requiring the user to pay attention to the slider and have their hand in a ready state to enable display of adjustment controls for the slider allows the computer system to automatically discern the user's intent to interact with the slider and avoid displaying additional controls when not needed.

It should be understood that the particular order in which the operations in fig. 13 are described is merely an example and is not intended to suggest that the order is the only order in which the operations may be performed. Those of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein with respect to other methods described herein (e.g., methods 8000, 9000, 10000, 11000, 12000, and 14000) apply in a similar manner to method 13000 described above with respect to fig. 13. For example, the user interface elements and regions (including their appearance and visual properties), virtual objects, user viewpoints, three-dimensional environments (including their views and appearances), display generation components, inputs, gestures, animations, user interface operations, light sources, reflections, and/or shadows described above with reference to method 13000 optionally have one or more of the characteristics of the user interface elements and regions (including their appearances and visual properties), virtual objects, user viewpoints, three-dimensional environments (including their views and appearances), display generation components, inputs, gestures, animations, user interface operations, light sources, reflections, and/or shadows described herein with reference to other methods described herein (e.g., methods 8000, 9000, 10000, 11000, 12000, and 14000). For the sake of brevity, these details are not repeated here.

FIG. 14 is a flow diagram of a method 14000 of visually distinguishing different portions of a computer-generated user interface object whose appearance is based on content in a surrounding three-dimensional environment behind the user interface object by applying different transformations to a representation of underlying content, according to some embodiments.

In some embodiments, the method 14000 is performed at a computer system (e.g., computer system 101 in fig. 1) in communication with a display generating component (e.g., display generating component 120 in fig. 1,3, and 4, such as display 7100 in fig. 7A-7Z) (e.g., heads-up display, touch screen, and/or projector) and optionally one or more input devices such as one or more cameras (e.g., a camera directed downward toward a user's hand (e.g., color sensor, infrared sensor, and/or other depth sensing camera) or directed forward from the user's head). In some embodiments, method 14000 is managed by instructions stored in a non-transitory computer-readable storage medium and executed by one or more processors of a computer system, such as one or more processors 202 of computer system 101 (e.g., control unit 110 in fig. 1A). Some operations in method 14000 are optionally combined and/or the order of some operations is optionally changed.

In some embodiments, method 14000 is performed at a computer system (e.g., computer system 101 in fig. 1, computer system 101 described with respect to fig. 7A-7G, 7H-7J, 7K-7O, 7P, 7Q-7W, and/or 7X-7Z). In some implementations, the computer system communicates with a display generating component (e.g., the first display generating component is a heads-up display, a head-mounted display (HMD), a display, a touch screen, a projector, etc.) and optionally one or more input devices (e.g., a camera, a controller, a touch-sensitive surface, a joystick, buttons, a glove, a watch, a motion sensor, and/or an orientation sensor). In some embodiments, the display generating component is the display generating component 7100 described with respect to fig. 7A-7Z. In some embodiments, the computer system is an integrated device having one or more processors and memory and at least some of the one or more input devices enclosed in the same housing as the display generating component. In some embodiments, the computer system includes a computing component (e.g., a server, a mobile electronic device such as a smart phone or tablet device, a wearable device such as a watch, wristband, earpiece, desktop computer, or laptop computer) that includes one or more processors and memory separate from the display generating component and/or one or more input devices. In some embodiments, the display generating component and the one or more input devices are integrated and packaged in the same housing. According to some embodiments, many of the features of method 14000 are described with respect to fig. 7X-7Z.

In method 14000, a computer system displays (14002), via a display generation component, a computer-generated user interface object overlaid on a first portion of a view of a three-dimensional environment (e.g., a view including one or more portions of a physical environment). The appearance of the computer-generated user interface object is based on the blurred representation of the first portion of the view of the three-dimensional environment covered by the computer-generated user interface object. For example, as described herein with reference to fig. 7X, the user interface element 7030 is overlaid on a view of the three-dimensional environment 7104 and has an appearance at time t=t ₂ that is based on a blurred representation of portions of the environment 7104 that are overlaid by the user interface element 7030.

In method 14000, a computer system visually distinguishes (14004) a first region of a computer-generated user interface object (e.g., text, type, glyph, etc.) from a second region of the computer-generated user interface object (e.g., other than text or type, such as a background region or one or more interactive controls) by: applying a first transformation to a first subset (e.g., a first set of one or more regions) of the blurred representation of the first portion of the view of the three-dimensional environment covered by the computer-generated user interface object, the first subset corresponding to the first region of the computer-generated user interface object, without applying a second transformation (e.g., different from the first transformation); and applying a second transformation to a second subset (e.g., a second set of one or more regions) of the blurred representation of the first portion of the view of the three-dimensional environment covered by the computer-generated user interface object without applying the first transformation, the second subset corresponding to the second region of the computer-generated user interface object. The second subset of blurred representations of the first portion of the view of the three-dimensional environment is different from the first subset of blurred representations of the first portion of the view of the three-dimensional environment.

For example, as shown in the expanded view 7084 of the appearance of the user interface element 7030 at time t ₂ and described herein with reference to this figure, in FIG. 7X, the text region 7084-d is visually distinguished from the background region 7084-a, because the text region 7084-d is the result of applying the transformation X _D but not the transformation X _A to portions of the starting point representation of time t ₂ that correspond to the text region 7084-d, And the background region 7084-a is the result of applying the transform X _A but not the transform X _D to the portions corresponding to the background region 7084-a, which are represented by the starting point of time t ₂. In another example, the text region 7084-d is visually distinguished from the control region 7084-b, because the text region 7084-d is the result of applying the transformation X _D but not the transformation X _B to portions of the starting point representation of time t ₂ that correspond to the text region 7084-d, And the control region 7084-b is a result of applying the transform X _B but not the transform X _D to the portions corresponding to the control region 7084-b, which are represented by the start point of the time t ₂. In yet another example, the text region 7084-c is visually distinguished from the background region 7084-a because the text region 7084-c is the result of applying an identity transform but not the transform X _A to portions of the starting point representation of time t ₂ that correspond to the text region 7084-c, and the background region 7084-a is the result of applying the transform X _A to portions of the starting point representation of time t2 that correspond to the background region 7084-a, rather than the identity transform.

In method 14000, as an appearance of a first portion of a view of the three-dimensional environment changes (e.g., in response to a change in the corresponding physical environment, a change in computer-generated content in the three-dimensional environment and/or overlaid on the corresponding physical environment representation, a change in a viewpoint of a user, or other change in viewing environment), the computer system updates (14006) the appearance of the computer-generated user interface object based on the blurred representation of the changed first portion of the view of the three-dimensional environment, including: applying (14008) a first transformation to a first subset of the blurred representation of the changed first portion of the view of the three-dimensional environment covered by the computer-generated user interface object, the first subset corresponding to a first region of the computer-generated user interface object, without applying a second transformation; and applying a second transformation to a second subset of the blurred representation of the changed first portion of the view of the three-dimensional environment covered by the computer-generated user interface object, the second subset corresponding to a second region of the computer-generated user interface object, without applying the first transformation. The second subset of blurred representations of the changed first portion of the view of the three-dimensional environment is different from the first subset of blurred representations of the changed first portion of the view of the three-dimensional environment.

For example, as described herein with reference to fig. 7Y, in accordance with changes in environment 7104, including changes in portions of environment 7104 covered by user interface element 7030, the appearance of user interface element 7030 at time t=t ₃ is updated based on the blurred representation of the changed portions of environment 7104 covered by user interface element 7030. For the appearance of the user interface element 7030 at time t ₃, as shown in the expanded view 7086 in fig. 7Y, the text region 7086-d is visually distinguished from the background region 7086-a, because the text region 7086-d is the result of applying the transform X _D but not the transform X _A to portions corresponding to the text region 7086-d represented by the starting point of time t ₂, While the background region 7086-a is the result of applying the transform X _A but not the transform X _D to the portions corresponding to the background region 7086-a, which are represented by the starting point at time t ₂. In another example, the text region 7086-d is visually distinguished from the control region 7086-b, because the text region 7086-d is the result of applying the transformation X _D but not the transformation X _B to portions of the starting point representation of time t ₃ that correspond to the text region 7086-d, while control region 7086-b is the result of applying transform X _B but not transform X _D to the portions of control region 7086-b represented by the starting point at time t ₃. In yet another example, the text region 7086-c is visually distinguished from the background region 7086-a because the text region 7086-c is the result of applying an identity transform but not the transform X _A to portions corresponding to the text region 7086-c represented by the starting point at time t ₃, and the background region 7086-a is the result of applying the transform X _A to portions corresponding to the background region 7086-a represented by the starting point at time t ₃ instead of applying the identity transform.

In some embodiments, the appearance of the first region is based on the blurred representation of the respective portion of the view of the three-dimensional environment covered by the first region to a greater extent than the appearance of the second region is based on the blurred representation of the respective portion of the view of the three-dimensional physical environment covered by the second region. In some embodiments, the first and/or second regions of the computer-generated user interface object are at least partially transparent (e.g., cut out from an otherwise solid shape) (e.g., a first transformation is applied to one or more cut-out regions and a second transformation is applied to one or more regions of the computer-generated user interface object that are outside the cut-out regions). Text, buttons, or control areas, and other areas such as background areas). Applying different transformations to different areas of a computer-generated object causes the computer system to automatically visually distinguish some areas from others to indicate certain attributes of those areas, such as whether the areas are interactive and/or whether the areas convey semantic meaning, and to visually emphasize and/or improve legibility of those areas accordingly.

In some embodiments, the appearance of the first portion of the view of the three-dimensional environment covered by the computer-generated user interface object changes in response to an environmental change in the three-dimensional environment (e.g., independent of movement of the user's point of view and independent of visual changes in movement of the computer-generated user interface object relative to the three-dimensional environment, such as due to movement of other people or objects in the three-dimensional environment or corresponding physical environment that are behind the simulated location of the computer-generated user interface object and/or due to changes in light in the three-dimensional environment or physical environment). For example, as described herein with reference to fig. 7X-7Y, the change in appearance of the user interface element 7030 is in some cases based on a change in appearance of one or more underlying portions of the environment 7104 due to movement of other objects in the environment 7104 and/or a change in illumination in the environment 7104. Changing the appearance of computer-generated objects based on environmental changes in a three-dimensional environment improves the user's situational awareness, which improves user safety by helping the user avoid collisions with physical objects in physical space, and reduces the likelihood that the user will suffer from motion sickness when using the computer system.

In some implementations, the appearance of the first portion of the view of the three-dimensional environment changes in response to the computer-generated user interface object moving relative to the three-dimensional environment (e.g., the computer-generated user interface object moves so as to be overlaid on a different first portion of the view of the three-dimensional environment). For example, as described herein with reference to fig. 7X-7Y, the change in appearance of the user interface element 7030 is based in some cases on movement of the user interface element 7030 relative to the environment 7104 that changes which portions of the environment 7104 are underlying the user interface element 7030. Changing the appearance of the computer-generated object as it moves relative to the three-dimensional environment and based on the blurred representation of the underlying portion of the environment improves the user's situational awareness, which improves user safety by helping the user avoid collisions with physical objects in physical space, and reduces the likelihood that the user will suffer from motion sickness when using the computer system.

In some embodiments, the view of the three-dimensional environment is visible from a respective viewpoint of the user via the display generating component (e.g., based on a location of the user or a location of the computer system or component of the computer system in the physical environment), and an appearance of the first portion of the view of the three-dimensional environment changes (e.g., moves) in the three-dimensional environment in response to the respective viewpoint of the user (e.g., the user moves itself and/or moves the computer system in the physical environment, thereby changing the view of the three-dimensional environment visible from the user's current viewpoint). For example, as described herein with reference to fig. 7X-7Y, the change in appearance of the user interface element 7030 is in some cases based on movement of the viewpoint of the user 7002 relative to the user interface element 7030 or relative to the environment 7104 that changes which portions of the environment 7104 are underlying the user interface element 7030 from the perspective of the user 7002. Changing the appearance of the computer-generated objects in the view as the viewpoint of the user changes relative to the three-dimensional environment and basing the appearance of the computer-generated objects on a blurred representation of the underlying portion of the three-dimensional environment improves the user's situational awareness, which improves user safety by helping the user avoid collisions with physical objects in physical space and reduces the likelihood that the user will suffer from motion sickness when using the computer system.

In some embodiments, the color saturation of the blurred representation of the first portion of the view of the three-dimensional environment is increased relative to the appearance of the first portion of the view of the three-dimensional environment that is not covered by the computer-generated user interface object. Increased saturation of the blurred representation (e.g., the starting point representation) of environment 7104 is described in more detail herein with reference to fig. 7X and similarly applies to fig. 7Y. In other words, the appearance of the first portion of the view of the three-dimensional environment has a first saturation metric, and the appearance of the computer-generated user interface object is based on a blurred representation of the first portion of the view of the three-dimensional environment, the appearance of the blurred representation having a second saturation metric that is greater than the first saturation metric. In some embodiments, in response to detecting an input that moves a computer-generated user interface object such that the computer-generated user interface object is no longer overlaid on a first portion of a view of the three-dimensional environment (e.g., for eliminating or closing the computer-generated user interface object, or for moving a computer-generated user to overlay input on a different, second portion of the view of the three-dimensional environment), the first portion of the view of the three-dimensional environment is displayed (e.g., redisplayed) as having a first saturation metric (e.g., in place of the computer-generated user interface object). When basing the appearance of a computer-generated object on a blurred representation of the underlying portion of the three-dimensional environment, increasing the saturation of the representation allows the computer system to automatically visually emphasize and improve the legibility of different content regions of the computer-generated object relative to the three-dimensional environment.

In some embodiments, applying the second transformation includes applying a gray filter to a blurred representation (e.g., a respective subset) of a respective portion of the view of the three-dimensional environment covered by the computer-generated user interface object (or more specifically, the second region of the computer-generated user interface object) (e.g., as described herein with reference to transformation X _A of fig. 7X-7Z). In some embodiments, the second transformation reduces saturation of the blurred representation of the corresponding portion of the view of the three-dimensional environment (e.g., reduces intensity of the color and/or increases the amount of gray). In some embodiments, some or all of the saturation increase of the blurred representation of the first portion of the three-dimensional environment described above is counteracted by a gray filter (e.g., a decrease in saturation) of the second transformation. Applying the gray filter to some areas of the computer-generated object whose appearance is based on the saturated-blurred representation of the underlying portion of the three-dimensional environment causes the computer system to automatically visually emphasize and improve legibility of other content areas of the computer-generated object where the gray filter is not applied.

In some embodiments, the first region of the computer-generated user interface object includes a glyph (e.g., a region occupied by letters, symbols, and/or other displayed characters), and the second region of the computer-generated user interface object includes a background region of the computer-generated user interface object (e.g., at least some regions of the computer-generated user interface object that are included outside of the region occupied by the glyph). For example, a "first region" optionally corresponds to text region 7084-c (fig. 7X) and similar text region 7086-c (fig. 7Y), while a "second region" optionally corresponds to background region 7084-a (fig. 7X) and similar background region 7086-a (fig. 7Y), where the two sets of regions are the result of applying mutually different sets of transforms (e.g., apply transform X _C for text regions 7084-c and 7086-c, apply transform X _A for background regions 7084-a and 7086-a). In another example, a "first region" optionally corresponds to text region 7084-d (fig. 7X) and similar text region 7086-d (fig. 7Y), and a "second region" optionally corresponds to background region 7084-a (fig. 7X) and similar background region 7086-a (fig. 7Y), where the two sets of regions are the result of applying mutually different sets of transforms (e.g., apply transform X _D for text regions 7084-d and 7086-d, apply transform X _A for background regions 7084-a and 7086-a). Applying a different transformation to the text region of the computer-generated object than to the background region of the computer-generated object causes the computer system to automatically visually emphasize the text region and improve legibility of the text region.

In some embodiments, the second region of the computer-generated user interface object includes a background of the computer-generated user interface object (e.g., outside the glyphs and outside the one or more interactive control regions) (e.g., background region 7084-a (fig. 7X) and similar background region 7086-a (fig. 7Y)), and the third region of the computer-generated user interface object includes one or more interactive control regions (e.g., buttons, switches, sliders, and/or input fields) (e.g., regions of the interactive control that are outside the region occupied by any glyphs of the interactive control) (e.g., control region 7084-b (fig. 7X) and similar control region 7086-b (fig. 7Y)). In some embodiments, the computer system visually distinguishes a third region of the computer-generated user interface object (e.g., from the first region and the second region) by: applying a third transformation (e.g., different from the first transformation and the second transformation) to a third subset of the blurred representation of the first portion of the view of the three-dimensional environment covered by the computer-generated user interface object (e.g., a third set of one or more regions) without applying the first transformation, and without applying a second transformation to the third subset of the blurred representation of the first portion of the view of the three-dimensional environment covered by the computer-generated user interface object (e.g., as part of operation 14002 of method 14000), the third subset corresponding to the third region of the computer-generated user interface object, wherein the third subset of the blurred representation of the first portion of the view of the three-dimensional environment is different from the first subset and the second subset of the blurred representation of the first portion of the view of the three-dimensional environment.

In some embodiments, the computer system visually distinguishes the third region of the updated computer-generated user interface object by: applying a third transformation to a third subset of the blurred representation of the changed first portion of the view of the three-dimensional environment covered by the computer-generated user interface object without applying the first transformation and without applying a second transformation to a third subset of the blurred representation of the changed first portion of the view of the three-dimensional environment covered by the computer-generated user interface object (e.g., as part of operation 14006 of method 14000), the third subset corresponding to a third region of the computer-generated user interface object, wherein the third subset of the blurred representation of the changed first portion of the view of the three-dimensional environment is different from the first subset and the second subset of the blurred representation of the changed first portion of the view of the three-dimensional environment. In some embodiments, applying the second transform to the respective subset includes reducing the brightness (e.g., darkening) of the respective subset, and applying the third transform to the respective subset includes increasing the brightness (e.g., brightening or brightening) of the respective subset.

As shown in fig. 7X, for example, and described herein with reference to that figure, in the appearance of user interface element 7030 at time t ₂, The appearance of the control region 7084-b is generated by: the brightening transform X _B is applied and the darkening transform X _A is not applied and the darkening transform X _D is not applied to the corresponding portion represented by the starting point of time t ₂, This visually distinguishes the background region 7084-a whose appearance is generated by applying the darkening transform X _A to the different corresponding portions of the starting point representation of time t ₂, but not the brightening transform X _B. Continuing with the same example, as shown in fig. 7Y and described herein with reference to that figure, in the appearance of the user interface element 7030 at time t ₃, as updated in response to changes in one or more portions of the environment 7104 under the user interface element 7030, The appearance of the control region 7086-b is generated by: the brightening transform X _B is applied to the corresponding portion of the changed starting point representation at time t ₃ without the dimming transform X _A and without the dimming transform X _D, This visually distinguishes the background region 7086-a whose appearance is generated by applying the darkening transform X _A to the different corresponding portions of the changed starting point representation of time t ₃, but not the brightening transform X _B. Applying a transformation that lightens the interactive control region of the computer-generated object, and alternatively applying a transformation that darkens the background region of the computer-generated object, visually emphasizes the region of the computer-generated object that interacts with respect to the non-interactive background region, which provides improved visual feedback regarding the state of the computer system.

In some embodiments, the degree to which the luminance of the respective subset is reduced when the second transform is applied to the respective subset is greater than the degree to which the luminance of the respective subset is reduced when the first transform is applied to the respective subset. In other words, a first region of the computer-generated user interface object (e.g., a region occupied by glyphs and/or text) is darkened less than a second region of the computer-generated user interface object (e.g., the first region is brighter than the second region) relative to the blurred representation of the underlying portion of the view of the three-dimensional environment. In some implementations, the first transform does not reduce the brightness of the respective subset (e.g., does not apply a darkening effect to the respective subset). For example, as shown in fig. 7X-7Y and described herein with reference to these figures, the text regions 7084-c and 7086-c are darkened to a lesser extent relative to their corresponding regions in the starting point representation than the background regions 7084-a and 7086-a are darkened to relative to their corresponding regions in the starting point representation by the transformation X _A (e.g., because no darkening transformation is applied to the text regions 7084-c and 7086-c).

In some embodiments, the degree to which the brightness of the respective subset is increased when the third transform is applied to the respective subset is greater than the degree to which the brightness of the respective subset is increased when the first transform is applied to the respective subset. In other words, the first region of the computer-generated user interface object is less bright than the third region of the computer-generated user interface object (e.g., the first region is darker than the third region) relative to the blurred representation of the underlying portion of the view of the three-dimensional environment. In some embodiments, the first transformation does not increase the brightness of the respective subset (e.g., does not apply a brightening effect to the respective subset). For example, as shown in fig. 7X-7Y and described herein with reference to these figures, the text regions 7084-c and 7086-c are not as bright as their corresponding regions in the starting point representation relative to the control regions 7084-b and 7086-b by the transformation X _B (e.g., because no brightening transformation is applied to the text regions 7084-c and 7086-c). The application of different transformations to different regions of a computer-generated object that cause the brightness of the text region to be higher than the background region and lower than the interactive control region allows the computer system to automatically improve the legibility of the text region while also providing improved visual feedback as to which regions are interactive.

In some embodiments, the degree to which the brightness of the respective subset is reduced when the second transform is applied to the respective subset is less than the degree to which the brightness of the respective subset is reduced when the first transform is applied to the respective subset. In other words, the first region of the computer-generated user interface object (e.g., composed of regions occupied by glyphs and/or text) is darkened to an even greater extent than the second region of the computer-generated user interface object (e.g., composed of background regions) relative to the blurred representation of the underlying portion of the view of the three-dimensional environment. For example, as shown in fig. 7X-7Y and described herein with reference to these figures, the text regions 7084-d and 7086-d are darkened by the transformation X _D relative to their corresponding regions in the starting point representation even less than the background regions 7084-a and 7086-a are darkened by the transformation X _A relative to their corresponding regions in the starting point representation. Applying different transformations to different regions of a computer-generated object that make the text region darker than other regions causes the computer system to automatically visually distinguish the text region from other regions in a manner that improves legibility of the text region.

It should be understood that the particular order in which the operations in fig. 14 are described is merely an example and is not intended to suggest that the order is the only order in which the operations may be performed. Those of ordinary skill in the art will recognize a variety of ways to reorder the operations described herein. In addition, it should be noted that the details of other processes described herein with respect to other methods described herein (e.g., methods 8000, 9000, 10000, 11000, 12000, and 13000) apply in a similar manner to method 14000 described above with respect to fig. 14. For example, the user interface elements and regions (including their appearance and visual properties), the blurred representation (including a subset thereof), the transforms, the virtual objects, the user viewpoints, the three-dimensional environment (including their views and appearances), the display generating components, and/or the inputs described above with reference to method 14000 optionally have one or more of the user interface elements and regions (including their appearances and visual properties), the blurred representation (including a subset thereof), the transforms, the virtual objects, the user viewpoints, the three-dimensional environment (including their views and appearances), the display generating components, and/or the characteristics of the inputs described herein with reference to other methods (e.g., methods 8000, 9000, 10000, 11000, 12000, and 13000) described herein. For the sake of brevity, these details are not repeated here.

The operations described above with reference to fig. 8, 9, 10, 11, 12, 13, and 14 are optionally implemented by the components depicted in fig. 1-6. In some embodiments, aspects and/or operations of methods 8000, 9000, 10000, 11000, 12000, 13000, and 14000 may be interchanged, substituted, and/or added between these methods. For the sake of brevity, these details are not repeated here.

The foregoing description, for purposes of explanation, has been described with reference to specific embodiments. However, the illustrative discussions above are not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in light of the above teaching. The embodiments were chosen and described in order to best explain the principles of the invention and its practical application to thereby enable others skilled in the art to best utilize the invention and various described embodiments with various modifications as are suited to the particular use contemplated.

Claims (25)

1. A method, comprising:

At a computer system in communication with a display generation component:

Simultaneously displaying via the display generating means:

A view of the physical environment; and

A computer-generated user interface element overlaid on at least a portion of the view of the physical environment, wherein an appearance of the computer-generated user interface element is based on an appearance of one or more portions of the view of the physical environment that the computer-generated user interface element overlays;

When the view of the physical environment and the computer-generated user interface element are displayed simultaneously, an appearance of a portion of the physical environment changes; and

In response to the appearance change of the portion of the physical environment, the appearance of the computer-generated user interface element is updated at a first time based on a graphical composition of appearances of the one or more portions of the physical environment at different times prior to the first time, the appearance of the one or more portions of the physical environment at different times prior to the first time including:

an appearance of a first portion of the physical environment at a second time prior to the first time; and

The appearance of the second portion of the physical environment at a third time prior to the second time.

2. The method of claim 1, wherein an appearance of the portion of the physical environment changes in response to movement of the computer system relative to the physical environment.

3. The method of claim 1, wherein the appearance of the portion of the physical environment changes in response to movement of one or more objects in the physical environment, wherein the one or more objects are different from the computer system.

4. The method of claim 1, wherein the computer-generated user interface element comprises content, and the updating of the appearance of the computer-generated user interface element based on the graphical composition is applied to one or more regions of the computer-generated user interface element that are different from the content of the computer-generated user interface element.

5. The method of any one of claims 1 to 4, wherein:

the graphical composition is based on a first amount of the appearance of the first portion of the physical environment at the second time before the first time and based on a second amount of the appearance of the second portion of the physical environment at the third time before the second time; and

The first amount is different from the second amount.

6. The method of claim 5, wherein at the first time, a contribution of one or more portions of the physical environment to the appearance of the computer-generated user interface element at the first time is greater than a contribution of the one or more portions of the physical environment to the appearance of the computer-generated user interface element at the first time at a time further from the first time.

7. The method of claim 6, wherein a contribution of one or more portions of the physical environment to the appearance of the computer-generated user interface at the first time at a time furthest from the first time is less than a contribution of the one or more portions of the physical environment to the appearance of the computer-generated user interface at the first time at any time closer to the first time.

8. The method of any of claims 1-4, wherein the graphical composition is based on the appearance of the one or more portions of the physical environment at a plurality of different times prior to the first time, and the plurality of different times is based on an amount of movement of the computer system relative to the physical environment.

9. The method of any of claims 1-4, wherein determining the graphical composition includes applying a noise pattern to the captured information about the appearance of the one or more portions of the physical environment at the different times.

10. The method of any of claims 1-4, wherein, in determining the graphical composition of the one or more portions of the physical environment at the different times, the appearance of the first portion of the physical environment at the second time is offset from the appearance of the second portion of the physical environment at the third time based on a difference between a perspective of the computer system at the second time and a perspective of the computer system at the third time.

11. The method of any of claims 1-4, wherein determining the graphical composition of the appearance of the one or more portions of the physical environment at the different times comprises obscuring a view of the physical environment captured by one or more cameras of the computer system.

12. The method of any one of claims 1 to 4, wherein the graphic composition is partially translucent.

13. The method of any of claims 1-4, wherein determining the graphical composition of the appearance of the one or more portions of the physical environment at the different times comprises reducing a resolution of information captured about the one or more portions of the physical environment at different times prior to the first time.

14. The method of any of claims 1-4, wherein the appearance of the computer-generated user interface element is updated as a viewing perspective of the computer-generated user interface element changes.

15. The method of any of claims 1-4, wherein the appearance of the computer-generated user interface element is updated as the computer-generated user interface element moves.

16. The method of any of claims 1-4, updating the appearance of the computer-generated user interface element based on one or more additional computer-generated user interface elements having a simulated location that is behind a simulated location of the computer-generated user interface element.

17. The method of claim 16, wherein the appearance of the one or more additional computer-generated user interface elements is updated over time.

18. The method of any of claims 1-4, wherein the computer-generated user interface element is a back surface of a displayed user interface comprising one or more selectable user interface objects.

19. The method of any of claims 1-4, wherein the appearance of the computer-generated user interface element is based on an appearance of one or more portions of the view of the physical environment covered by the computer-generated user interface element regardless of a simulated position of the computer-generated user interface element relative to the physical environment.

20. The method of any of claims 1-4, wherein the simulated location of the computer-generated user interface element is a first distance from a viewpoint of a user, and the appearance of the computer-generated user interface element is based on an appearance of one or more portions of the physical environment that are a second distance from the viewpoint of the user, wherein the first distance is greater than the second distance.

21. A computer system in communication with a display generation component, the computer system comprising:

One or more processors; and

A memory storing one or more programs, wherein the one or more programs are configured to be executed by the one or more processors, the one or more programs comprising instructions for:

Simultaneously displaying via the display generating means:

A view of the physical environment; and

A computer-generated user interface element overlaid on at least a portion of the view of the physical environment, wherein an appearance of the computer-generated user interface element is based on an appearance of one or more portions of the view of the physical environment that the computer-generated user interface element overlays;

When the view of the physical environment and the computer-generated user interface element are displayed simultaneously, an appearance of a portion of the physical environment changes; and

In response to the appearance change of the portion of the physical environment, the appearance of the computer-generated user interface element is updated at a first time based on a graphical composition of appearances of the one or more portions of the physical environment at different times prior to the first time, the appearance of the one or more portions of the physical environment at different times prior to the first time including:

an appearance of a first portion of the physical environment at a second time prior to the first time; and

The appearance of the second portion of the physical environment at a third time prior to the second time.

22. The computer system of claim 21, wherein the one or more programs include instructions for performing the method of any of claims 1-20.

23. A computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computer system in communication with a display generation component, cause the computer system to perform operations comprising:

Simultaneously displaying via the display generating means:

A view of the physical environment; and

A computer-generated user interface element overlaid on at least a portion of the view of the physical environment, wherein an appearance of the computer-generated user interface element is based on an appearance of one or more portions of the view of the physical environment that the computer-generated user interface element overlays;

When the view of the physical environment and the computer-generated user interface element are displayed simultaneously, an appearance of a portion of the physical environment changes; and

In response to the appearance change of the portion of the physical environment, the appearance of the computer-generated user interface element is updated at a first time based on a graphical composition of appearances of the one or more portions of the physical environment at different times prior to the first time, the appearance of the one or more portions of the physical environment at different times prior to the first time including:

an appearance of a first portion of the physical environment at a second time prior to the first time; and

The appearance of the second portion of the physical environment at a third time prior to the second time.

24. The computer readable storage medium of claim 23, wherein the one or more programs comprise instructions, which when executed by the computer system, cause the computer system to perform the method of any of claims 2-20.

25. A computer program product comprising instructions which, when executed by a computer system, cause the computer system to perform the method of any of claims 1-20.