Have published a prebuilt builder image at https://hub.docker.com/repository/docker/fredrikfornwall/package-builder-r28b-v1, which can be used to save time as:

CONTAINER_NAME=ndk28-builder TERMUX_BUILDER_IMAGE_NAME=fredrikfornwall/package-builder-r28b-v1 ./scripts/run-docker.sh

We've only been updating to the LTS NDKs in recent years.

We've only been updating to the LTS NDKs in recent years.

Ah right, r25, r26 and r27 were three consecutive LTS releases.

I wonder if it makes sense to continuously update to NDK stable releases without awaiting LTS ones, to more continuously adapt to toolchain changes as soon as they are out?

Honestly, their release process is a mess at this point, is "stable" still considered a rolling release? Who knows.

I definitely think it makes sense to adapt to the stable releases in a branch, but whether we should switch the whole repo over now is an open question. Maybe Henrik, @truboxl, @licy183, and other contributors have input...

I'd suggest that we wait for r28c. Regardless of r28c being LTS or not, I guess we should go ahead with it

I'm afraid NDK r28* will not be the LTS release... See https://github.com/android/ndk/wiki

I'm afraid NDK r28* will not be the LTS release

Yes, we know it won't be: Fredrik is proposing that we switch to it anyway.

Updated to the newly released r28c.

@thunder-coding (and others): What do you think, should we go ahead and merge this?

I prefer to wait until 2025-07-16

Wrong button, thanks GitHub mobile app.

should we go ahead and merge this?

If it's working well to build packages, no problem.

I prefer to wait until 2025-07-16

How come?

Updated to the newly released r28c.

@thunder-coding (and others): What do you think, should we go ahead and merge this?

Yeah, sure go ahead! Let's agree on a time first as lot of rebuilds need to be done. I'll handle nodejs, nodejs-lts, ccls and libicu, their dependencies and revdeps

Let's agree on a time first as lot of rebuilds need to be done

Why is a lot of rebuilds necessary - is it to modify the build / add patches to adopt them to build with NDK r28 (and meanwhile, the existing builds won't be broken at runtime, right)?

Existing packages which depend on libc++ will be broken at runtime since this release of NDK updates the LLVM version as well just like other major releases of NDK.

Existing packages which depend on libc++ will be broken at runtime since this release of NDK updates the LLVM version as well just like other major releases of NDK.

As I understand it libc++ is backward compatible - it does not break ABI compatibility, so there is no need to rebuild everything after updating libc++ to a newer version. https://libcxx.llvm.org/DesignDocs/ABIVersioning.html:

Libc++ aims to preserve a stable ABI to avoid subtle bugs when code built under the old ABI is linked with code built under the new ABI. At the same time, libc++ wants to make ABI-breaking improvements and bugfixes in scenarios where the user doesn’t mind ABI breaks.
To support both cases, libc++ allows specifying an ABI version at build time. The version is defined with CMake option LIBCXX_ABI_VERSION. Currently supported values are 1 (the stable default) and 2 (the unstable “next” version). At some point “ABI version 2” will be frozen and new ABI-breaking changes will start being applied to version 3; but this has not happened yet.

As I understand it libc++ is backward compatible - it does not break ABI compatibility, so there is no need to rebuild everything after updating libc++ to a newer version. libcxx.llvm.org/DesignDocs/ABIVersioning.html:

Hmm.. Atleast earlier I recall NDK updates requiring full rebuilds of almost everything, maybe things have changed in newerer releases.

Ref: #17863 (comment)

I'll try to test some packages out in termux-docker this weekend if you are not in a hurry to merge this

Shortly after the NDK r28c is merged, I can begin a build validation using the build-validation.sh from the pinned issue, so I can update the list in the pinned issue to add packages that will need to be patched to build with CMake 4.0, NDK r28c, 16 KiB page size, or any of the newer Clang and Clang++ and C and C++ features that are most likely added in NDK r28c

I understand this might not be a big problem, but I would just like to put a note somewhere that the Rust documentation has a comment that slightly implies the latest LTS NDK might be required for Rust on Android https://doc.rust-lang.org/rustc/platform-support/android.html#building-the-target

To build Rust binaries for Android you'll need a copy of the most recent LTS edition of the Android NDK

Whether NDK r28c specifically would be OK is not mentioned, but if nothing thunder-coding tested had problems, then I assume there aren't any major problems.

I prefer to wait until 2025-07-16

How come?

Thanks for waiting. 2025-07-16 is the end date for r27c support.
It looks like the next LTS NDK is r27d.
The next one after may be r27e. NDK r27e
I think we can break the LTS NDK rule this time.

Why is a lot of rebuilds necessary - is it to modify the build / add patches to adopt them to build with NDK r28 (and meanwhile, the existing builds won't be broken at runtime, right)?

Full rebuild is necessary this time around. Need to make sure all packages meet #21688.
https://developer.android.com/guide/practices/page-sizes#alignment-use-tools suggests to use llvm-objdump -p to scan binaries and libs that do not have 2**14 in their LOAD segments. Most binaries I scanned are 2**12. So not compulsary to set up emulator to test.

I like to add r27c already shipped libc++_shared.so with 2**14 and we havent received any reports that cause problems with C++ packages that are still 2**12.

Full rebuild is necessary this time around

Right, we need to rebuild everything to have 16 kb page sizes - but we don't need to rush it and rebuild everyhing in a few days, right? Because everything should still work during a mixed transition period (where some packages have 4 kb page size, and some 16 kb)?

16KB builds should be backward compatible, and no immediate hurry, but should be done in coming weeks/months so that it can be tested. At least bootstrap zip packages and their dependencies should be rebuilt first, I should be able to find the list from bootstrap build logs.

Does all that mean that if the app does target Android 15 or 16 and doesn't support 16KB alignment. It won't work on it? Or will it just work kinda in a backwards compatible mode? Did anyone test this. And what if the app bundles a compressed file like the termux bootstrap then the Android (and GooglePlay) will think that the app does support 16kb alignment because it didn't detect and native binaries at all.
Does that mean that the app will crash when the user is trying to run a binary or not?

In app native libraries and all binaries in bootstrap need to be recompiled for 16KB, otherwise will cause a crash. Target sdk version does not matter.

Hmm so what about the packages built on device? (Using clang, rust. etc.)
Will they need a max-page-size as well?
If they will then how will the be achieved? Maybe by patching llvm and do something like https://reviews.llvm.org/D55029
?? Or is this not needed at all?
EDIT: Seems like no patching is needed
clang already changes the max-page-size to 16kb on Android 64bit.
https://github.com/llvm/llvm-project/blob/f79efa986d61700d3fcfd22390bc1aa17d0d454c/clang/lib/Driver/ToolChains/Linux.cpp#L238-L243

Good question. Likely required flags for 16KB will need to be passed for all compilers if they are not being passed by default.

It seems like everything should work out of the box (at least for clang)
Because of this commit
llvm/llvm-project@9a28c86

So what should we do now?
Try to just rebuild most packages and see if there are errors?
i believe this is a high priority cause it seems that this breaks termux and termux-packages for (most) Android 15 & 16 devices (Btw i didn't test anything :/)

Also a bit off topic but what about old apps that uses native code and didn't get any updates.. Does that mean that they will just break and that's it? Is there is no other way around??

This 16 KB issue is not a high priority. The problem only occurs on Android devices that have a kernel compiled with 16 KB pages, which almost no devices are likely using yet. Native code that has the new 16 KB alignment works everywhere, ie on older 4 KB kernel devices also, while most Android 15 and 16 devices will likely continue having 4 KB kernels.

We should switch all our native code to 16 KB in the coming year, but the problem is not going to hit many devices any time soon.

Yeah, I doubt existing devices would receive an OS upgrade with 16KB ROM, as that would break the already installed apps if they don't support 16KB native JNI libraries or binaries. Likely future devices that are newly released with a fresh android 15/16+ install may start with 16KB, which would require termux app+bootstrap and also all/popular packages in repo to support 16KB, and on such devices users will need to start from a fresh termux install and won't be able to restore backups from other/older devices of prefix or other binaries in $HOME that were compiled for 4KB.

Since the topic has come up, I guess it might be relevant to mention that I have a suspicion that termux-docker on Asahi Linux might fail to work unless its docker image is built from a future 16kb-compatible Termux bootstrap. However, I'm only about 65% sure of this because, I don't have regular access to any Macs to test that, and nobody who has Asahi Linux (bare metal Apple Silicon GNU/Linux with Docker installed and running containers on a kernel that runs bare metal on the Apple hardware) has mentioned what happened if they tested the current termux-docker.

Here is a related issue for another Android-in-docker container, Redroid:

• mprotect error on ARM64 16k pages host remote-android/redroid-doc#849

For the case of redroid, if any 4k pagesize problems really are reproducible inside Termux inside an Android image inside Docker on the kernel of an Asahi Linux device, those would be reproducible by, for example, installing the Termux APK inside redroid.

(Note: I skimmed over some details here to make the explanation simpler, to the point, and not create a huge text wall).

I skimmed over some details here to make the explanation simpler, to the point, and not create a huge text wall

You appear to be unaware of the main mode of communication here, particularly of some of the lead devs. 😝 More seriously, sometimes we have no alternative, being a purely online org, and when it is a pertinent comment full of excellent technical and sourced info like the one I linked, nobody can ever complain.

In your case, Termux on Asahi is such a niche use case that the pertinence requirement is not fulfilled, so it's good you abbreviated, but definitely relevant that you mentioned it as a possible test case for this new requirement.