Can I help you with this project?

I am a programmer with very little Java experience, but I am a Linux power user and work as a Linux sysadmin; I also have a Quest 2

I also wanted to ask/suggest possibly leveraging wxrd (https://gitlab.freedesktop.org/xrdesktop/wxrd), a lightweight Linux VR compositor based on wlroots, or alternatively xrdesktop-gnome-shell, or xrdesktop-kdeplasma. Unsure if this helps or hinders your development, but these compositors provide a 3D VR environment with free-floating app windows, and controller support

Can I help you with this project?

Let me make the basic integration working first. Currently it is just a black screen and it does nothing.
I will ping you once I have something working what could be improved.

I also wanted to ask/suggest possibly leveraging wxrd (https://gitlab.freedesktop.org/xrdesktop/wxrd), a lightweight Linux VR compositor based on wlroots, or alternatively xrdesktop-gnome-shell, or xrdesktop-kdeplasma. Unsure if this helps or hinders your development, but these compositors provide a 3D VR environment with free-floating app windows.

I didn't know about xrdesktop, it looks pretty wild. It would be ride to make it working on standalone. I imagine it quite challenging to made that working on Quest but I might be wrong.

I also wanted to ask/suggest possibly leveraging wxrd (https://gitlab.freedesktop.org/xrdesktop/wxrd), a lightweight Linux VR compositor based on wlroots

Termux:X11 is not related to Wayland.

@twaik

I also wanted to ask/suggest possibly leveraging wxrd (https://gitlab.freedesktop.org/xrdesktop/wxrd), a lightweight Linux VR compositor based on wlroots

Termux:X11 is not related to Wayland.

It was my understanding that Termux:X11 is an xwayland session. Weston reportedly works, and that is Wayland-based.

@lvonasek
Further projects that may prove useful testing these efforts; mesa-zink with turnip and/or virglrenderer are termux-viable projects which enable hardware 3D acceleration

The xrdesktop project also has Gnome and KDE-specific builds that are x11-based (https://gitlab.freedesktop.org/xrdesktop). The wxrd window manager was created to have an extremely small footprint. In all the aforementioned cases, xrdesktop is the underlying platform, which already has movement tracking and controller support. (Hoping "Direct Input" option for touchscreen passthrough could work to pass the controllers and head tracking to Monado without much trouble)

@beef-ox It is nice to see there are many opportunities. But until I have the basic integration working, I won't distract myself with other possible stuff. Key to success is to do small steps and do them properly.

It was my understanding that Termux:X11 is an xwayland session

It was first few years. Termux:X11 implemented a small subset of wayland protocol only to make it possible to run Xwayland. But at least year ago project dropped it because of architecture restrictions.

Weston reportedly works, and that is Wayland-based.

Weston works on top of X11 session. It does not need Wayland session to work, it starts wayland session.

The wxrd window manager was created to have an extremely small footprint

wxrd requires wlroots, which requires GLES with some extensions which can not be implemented on Android. Android vendors do not implement support for these extensions and even if they do they are not a part of SDK/NDK and not guaranteed to work.
It is a no go.

Hoping "Direct Input" option for touchscreen passthrough could work to pass the controllers and head tracking to Monado without much trouble

You have illusions about how that works. It is implemented only for touchscreen and passes only touchscreen events.

Termux:X11 does not use C++ to keep APK size as small as possible. Currently I am not intended to merge C++ code, only C.

Termux:X11 does not use C++ to keep APK size as small as possible. Currently I am not intended to merge C++ code, only C.

Ok, good to know. I will move the XR code to C.

There are a few more things:

1. You are using GLES3. Currently renderer uses GLES2, and I want to avoid mixing GLES versions in one project.
2. It seems like you are considering to use swapchain, which will mean blitting image from one frame to another. This solution will have less performance than the main code. Do you receive Surface or SurfaceFrame from OpenXR which will be used to draw on it? I think the best solution will be taking this Surface of SurfaceFrame and pass it directly to X server. But I am not sure how exactly this works.
3. I can add support for physical gamepad/controller/joystick, but I do not have device (gamepad) for tests. I do not play games that require that. But I can buy one in the case anyone sends funds to buy this (yeah, I buyed one from aliexpress, but it was a piece of ■■■■ and my devices did not even recognized its events correctly).

1. You are using GLES3. Currently renderer uses GLES2, and I want to avoid mixing GLES versions in one project.

I believe I can move to GLES2 completely. GLES3 would be needed if I use stereoscopical rendering using multiview extension.

2. It seems like you are considering to use swapchain, which will mean blitting image from one frame to another. This solution will have less performance than the main code. Do you receive Surface or SurfaceFrame from OpenXR which will be used to draw on it? I think the best solution will be taking this Surface of SurfaceFrame and pass it directly to X server. But I am not sure how exactly this works.

The swapchain is required by OpenXR. The only way to render in OpenXR is to render into texture and then let headset reproject it. This architecture is very helpful in VR as you can get fluent experience even when not rendering lower framerate than headset's refresh rate. In 2D rendering it doens't bring much benefit but it still needs to be used.

3. I can add support for physical gamepad/controller/joystick, but I do not have device (gamepad) for tests. I do not play games that require that. But I can buy one in the case anyone sends funds to buy this (yeah, I buyed one from aliexpress, but it was a piece of ■■■■ and my devices did not even recognized its events correctly).

I would like to avoid mapping Meta Quest touch controllers thumbsticks to joystick. The thumbsticks are getting after some time extreme noise. In other XR projects I check if the stick is 70% on right and if so then I send event to right key arrow. But of course we could make it optional at some point.

@lvonasek I am not really sure how exactly it works. How exactly you are intending to extract frames in the activity process? Currently LorieView (which works in context of MainActivity) does not output anything to Surface. It simply passes this Surface to X server process via Binder and X server (which works in com.termux's application sandbox) does all the magic. Of course you can use SurfaceTexture for this, but this solution will use more resources because X root window will be rendered one more time.

First, I need to figure out how the rendering in this project works.

Ideally, I would call glBindFramebuffer (binding my XrFramebuffer) and render the frame using OpenGL into it. That way the frame is in OpenXR. In OpenXR, I'll define I want to render it on a plane in 3D space.

It is work in progress and I am new to this repo, please be patient if I commit or say something stupid.

First, I need to figure out how the rendering in this project works.

I explained where exactly Surface is being used so I can explain rendering process too.
Renderer is pretty much simple. Right after initialisation of itself X server starts renderer initialisation. It gets jmethodIDs of some Surface related functions, prepares GLDisplay and primitive GLContext and does some checks (to determine if device supports sampling AHardwareBuffer in RGBA and BGRA formats) and creates AHardwareBuffer for root window (if it is supported). After initialisation X server waits for Activity connection. When activity is connected it sends Surface and related data. Renderer initialises new GLContext based on this Surface (in ANativeWindow shape), creates shader and textures for root window and cursor and allows X server draw on there. When server wants to draw screen or cursor is being changed/moved renderer uses shader to draw both root window and cursor textures on current GLSurface and invokes eglSwapBuffers.
In the case if device supports sampling AHardwareBuffer of required type root window texture is created with eglGetNativeClientBufferANDROID+eglCreateImageKHR+glEGLImageTargetTexture2DOES, otherwise it is created with simple glTexImage2D and being updated with glTexSubImage2D.
Cursor texture is updated with glTexImage2D because I did not meet animated hi-res cursors. But that can be fixed.

Actually this process is pretty much simple. You can reimplement the whole thing in pure vulkan and integrate it to your OpenXR related code.

But. I am not sure why OpenXR context is initialized with JavaVM and global reference of Activity. So I am not sure if it can run completely in X server process. I think I will understand it better in the case you elaborate how exactly that works.

I explained where exactly Surface is being used so I can explain rendering process too.
...

Thank you, this is very helpful.

But. I am not sure why OpenXR context is initialized with JavaVM and global reference of Activity. So I am not sure if it can run completely in X server process. I think I will understand it better in the case you elaborate how exactly that works.

For JavaVM I found nowhere any info why is it required. The activity itself is needed for app lifecycle (listening to onWindowFocusChange, onPause and onResume events). I try to elaborate but I am really not good at explaining:

AR/VR headsets have two app modes: 2D (Android apps flying in 3D space) and immersive OpenXR mode. In immersive mode the app cannot render anything using Android API. The only way to show something on screen is OpenGL/Vulkan. Meta recently added support for hybrid apps where you can switch between 2D and XR activity.

I added hybrid app support into this PR and trigger OpenXR runtime only if the app is running on a headset. The final APK will run on regular Android and XR headset(s). Currently it is under construction but in the future I would like to start XR only if the XServer is running (currently there is no way in the headset to go into the preferences or open help page).

• XrEngine takes care if OpenXR initialization. it defines which extensions are used and starts the immersive mode.
• XrRenderer is kind of 3D compositor. There are several layers in 3D space. The 3D space I am using is horizontally aligned with the floor level. The XrRenderer gets information about head`s motion tracking - XrPose (rotation using IMU sensor and relative position in meters using SLAM). All the magic is happening in the implementation of the headset. I just define, I want to render a geometrical shape with a specific dimensions in the space and map on it a texture in XR format (the mapping could be done separately for each eye).
• XrFramebuffer in a very vague words connects OpenGL texture to OpenXR texture handle. In the beginning of frame I acquire the OpenXR texture handle, bind OpenGL framebuffer, render into it and release it. Once the OpenXR texture handle is released, I could pass it to XrRenderer and that will show it to the user (without calling eglSwapBuffers, no Surface is used). To be fair there exists xrCreateSwapchainAndroidSurfaceKHR but I never saw any project using it (I would give that a try if there is no better option).
• XrInput provides status of the headset`s controllers (XrPose and buttons) and controls their vibrations.
• XrMath are just math utils which every developer has to implement because OpenXR math structs doesn`t contain any basic math operations.

@lvonasek

With all due respect, I would rather not lose the ability to render as a 2D app on the Quest's home launcher when the x server is displaying 2D content. There should be no need to do that.

x11 is a very important and well-understood protocol. If you want to implement Quest support, I don't think you should be creating a custom, made by you 3D environment to reproject onto, from which all further users of Termux:x11 will then be forced into using, over the Quest's multi-tasking launcher which lets you have 3 2D apps side by side; perfect for my programming workflow for example (and many others)

The goal of Tx11 should be to implement as much of the x11 client protocol as possible, and as close to spec in all respects as possible. The distinction as to whether it should attempt 2D mode vs immersive mode should not be reliant upon the device it is on, but upon whether the x server is attempting to display OpenXR content AND the hardware supports it.

I 100% agree, if the Linux environment is trying to output stereoscopic content over x11, this should indeed display it in immersive mode, but if not, it should display it as a 2D app window. Ideally, this could work like full screen, where the rendering pipeline is direct vs going through a compositor. 2D content displays in a traditional desktop "display" as a 2D app within a WM/DE, but attempting to display XR content would switch to immersive mode to display that content.

With all due respect, I would rather not lose the ability to render as a 2D app on the Quest's home launcher when the x server is displaying 2D content. There should be no need to do that.

I will definitely try to make that optional.

I am not skilled enough to check your C/Java OpenXR code, but I think I can review the Java part.

1. Is there a reason to have XrActivity::isEnabled if there is XrActivity::isSupported? I mean do we really need a wrapper?
2. In dispatchKeyEvent you check only KeyEvent.ACTION_UP action, and not checking KeyEvent.ACTION_DOWN at all. Also you compare system time of event. Probably you can check the KeyEvent::getDownTime and check if it is the same as one before this. I am pretty sure 4 consequent Enter KeyEvents will have the same down time. Also you can check the KeyEvent::getRepeatCount and ignore repeats.
3. About GLUtility::drawTexture. You generate ByteBuffer of constant data on every frame. I am pretty much sure it generates a lot of objects you abandon right after function returns. Consider creating pre-allocated ByteBuffer instance with static modifier.

The rest of code seems to be fine.

Thank you for the review.

1. That's for a follow-up PR to add into the settings an option to disable OpenXR even when it is supported. Someone here asked for it.

2. The headset's software keyboard sends only UP event. This way it will keep working correctly in case OS update adds a proper implementation. I will try to avoid using system time there.

3. Good catch, I will fix that.

Any another merge blocker @twaik?

I do not think so. I'll check it later.

Don't you mind if I squash all the commits into one?
47 commits is too much.

I am fine with squashing it.

Sorry for noob question, I managed to run XFCE4 on Q2 as 2d window, but when I switch to "oculus mode" just wait spinner (oculus 3 dot) continuously runs. What can I run in XR mode?

Sorry for noob question, I managed to run XFCE4 on Q2 as 2d window, but when I switch to "oculus mode" just wait spinner (oculus 3 dot) continuously runs. What can I run in XR mode?

Without log from logcat I cannot say much. Maybe an older OS version? (AFAIK it should work on V62 and higher)

The XR mode is mostly usable for gaming (using "mobox").

Sorry for noob question, I managed to run XFCE4 on Q2 as 2d window, but when I switch to "oculus mode" just wait spinner (oculus 3 dot) continuously runs. What can I run in XR mode?

Without log from logcat I cannot say much. Maybe an older OS version? (AFAIK it should work on V62 and higher)

The XR mode is mostly usable for gaming (using "mobox").

Do you mean > Mobox is a project designed to run windows x86 applications in Termux?
Than XFCE4 (any X window) must be also visible as 2d window in 3d? If yes will try to catch tx11 logs, Im on V66.
P.S. Im new to oculus, and it is the only one my android device without root, this is sad.

Do you mean > Mobox is a project designed to run windows x86 applications in Termux?

Yes

Than XFCE4 (any X window) must be also visible as 2d window in 3d?

Yes but without any advantages.

I did a similar integration like this for Winlator project: https://youtu.be/c4faL1G1St4?feature=shared

Cool! XFCE4 works in both modes, howeever. No way to return from preferencies activity, pressing/clicking/touching BACK buttons does nothing (in both modes). Have to close tx11 and open again to apply changes.
In Winlator I did not find how to switch to 2D at all

If you open system notifications, there is tx11 and from there you can open preferences.

I stopped working on Winlator because it isn't fully opensource.

If you open system notifications, there is tx11 and from there you can open preferences.

Ofcourse, but question is - how to close preferences

With the X button. However TX11 might need a restart if you change XR mode when running.

@lvonasek
Can you please tell me why you are doing this?

    private static int getMainDisplay(Context context) {
        final DisplayManager displayManager =
                (DisplayManager) context.getSystemService(Context.DISPLAY_SERVICE);
        for (Display display : displayManager.getDisplays()) {
            if (display.getDisplayId() == Display.DEFAULT_DISPLAY) {
                return display.getDisplayId();
            }
        }
        return -1;
    }

Display.DEFAULT_DISPLAY is always a display id you are looking for...

I mean you can use it directly.

I just followed this:
https://github.com/amwatson/2DVrHybrid

I will create a PR cleaning it up.

I am pretty much sure the

        // 1. Locate the main display ID and add that to the intent
        final int mainDisplayId = getMainDisplay(context);
        ActivityOptions options = ActivityOptions.makeBasic().setLaunchDisplayId(mainDisplayId);

is not necessary, but only you can check if that is true.

Can you please find out how to get out of 3d mode programmatically? Starting MainActivity and finishing XrActivity does not seem to work.

I mean

getBaseContext().startActivity(new Intent(this, MainActivity.class)
        .addFlags(Intent.FLAG_ACTIVITY_NEW_TASK | Intent.FLAG_ACTIVITY_CLEAR_TASK));
finish();

I will take a look

Hi
I get this screen tearing or whatever in quest 3 v67
in both xr mode and 2d
though in xr mode it happens only when its not in fullscreen
Could you have a look please

No, this isn't connected to my implementation. It seems something to do with a GPU driver. Report it to Mobox or whatever you use.

Hi
If its something to do with a GPU driver it shouldn't work in xr mode .it does
I think it related to scaling (resize) a window it start to behave like that, in fullscreen its good
In 2d mode its broken
Also is the feature moving screen closer or farther possible in xr mode cause its fixed
Thanx

If its possible to move screen in xr mode that solves my problem

It is a GPU problem. XR renders the data into an own framebuffer and then on a screen. I guess that makes the difference.

I stop doing support in this PR. All further messages will be ignored.

Sorry for wasting your time with🙏 this
All I want to know if it plausible to move and resize screen in xr mode
Thanx