bump(main/pypy3): 7.3.19 #24525

robertkirkman · 2025-04-28T06:38:58Z

Caution

downside: build-time performance regression

for the ARM targets, the current pypy3 build.sh in the master branch has a build-time performance optimization involving the use of a double proot with the first proot being an x86 bionic-libc proot for all targets and the nested proot inside the first proot being the only one which requires the qemu-user emulator for the ARM targets. Unfortunately, termux-proot-run does not implement that optimization, so the loss of the optimization increases the build time of pypy3 for ARM targets. This is not good, so this PR can be disregarded, or replaced with a more conservative one, or a better-optimized one, if the build-time performance regression is considered really undesirable.

potential replacement of pypy3: bump to 7.3.17 #21297
- contains a do-not-try-to-get-L2-cache-on-aarch64.patch copied from it
continuation of scripts(build-packages.sh,build): Add support to run target binaries #23966 (comment)
bump TERMUX_PKG_VERSION to 7.3.19 (this number indicates the PyPy-specific release version)
bump _MAJOR_VERSION to 3.11 (this number indicates PyPy's compatibility level with Python code written for CPython)
make build.sh and .patch files shorter by removing a lot of things and replacing them with ways that have fewer lines
- deletes:
  - 0006-package-zip.patch
  - 0007-termux-build.patch
  - 0008-do-not-cffi-dlopen-when-compiling-sqlite3.patch
  - 0010-dummy-cpu_count.patch
  - cc.sh
  - termux.py.in
- because I did something different that does not seem to require them.
- however if deleting any of these is incorrect, please tell me and I could try to put them back.
deblobbify the pypy3 build process by removing this line and replacing it with a total migration to termux-proot-run instead of a termux-docker image containing blobs like this one
remove build dependencies on coreutils, clang, make and pkg-config and replace them with a build dependency on dnsmasq
- this is caused by the way that the migration to termux-proot-run is implemented
pypy3/0008-fix-ctypes-find_library.patch and python2/fix-ctypes-find_library.diff:
- fix OSError: ctypes.util.find_library() did not manage to locate a library called 'sqlite3' which started appearing, most likely after libsqlite: bump to 3.49.1 #23556
- (0008-do-not-cffi-dlopen-when-compiling-sqlite3.patch, as it is in the master branch currently, is not able to prevent this error, at least as far as I could tell)
fix postinst and prerm scripts
- add postinst script to correctly allow migrating the _MAJOR_VERSION
- adjust prerm scripts to fix a problem with $TERMUX_PACKAGE_FORMAT=pacman mode, by @Maxython, copied and pasted from tree-wide: port debpython to termux #23652
implements $TERMUX_ON_DEVICE_BUILD=true compatibility
- (requires a $TERMUX_ON_DEVICE_BUILD=true-compatible device with a large amount of memory. Tested using a device with 16 GB of RAM and 0 GB of swap.)
On February 26th, one day after we wrote set-_FILE_OFFSET_BITS-to-64-on-arm.patch: pypy: bump to 7.3.18 #23474 (comment), upstream discovered, reported, independently also bisected, and on March 24th, independently also fixed the same problem using code that is character for character exactly identical to our patch: Translation broken on 32-bit ARM on older platforms: assert OFF_T_SIZE == rffi.sizeof(rffi.LONGLONG) pypy/pypy#5229, stefanor/pypy@7509998

I tried to also apply the below change to pypy3 like what was done to pypy (perform the rpython translation-step using a GNU/Linux pypy2 instead of a bionic-libc python2)

https://github.com/termux/termux-packages/pull/21026/files#diff-7aabb4b13f8da30e9e862e34c866d5b862ba8c07358d4a126a5c40362dc4b8e0R36-R50

however I could not successfully figure out how to do it. I was able to proceed past a few errors, but I kept getting stuck on this error. The externmod.so is a bionic-libc artifact, and I can't figure out how to get it to work in the GNU/Linux pypy2.

error from failed attempts to build using a GNU/Linux PyPy2

[translation:info] Error:
   File "/home/builder/.termux-build/pypy3/src/rpython/translator/goal/translate.py", line 286, in main
    default_goal='compile')
   File "/home/builder/.termux-build/pypy3/src/rpython/translator/driver.py", line 584, in from_targetspec
    spec = target(driver, args)
   File "targetpypystandalone.py", line 371, in target
    return self.get_entry_point(config)
   File "targetpypystandalone.py", line 410, in get_entry_point
    self.space = make_objspace(config)
   File "/home/builder/.termux-build/pypy3/src/pypy/tool/option.py", line 33, in make_objspace
    return StdObjSpace(config)
   File "/home/builder/.termux-build/pypy3/src/pypy/interpreter/baseobjspace.py", line 457, in __init__
    self.initialize()
   File "/home/builder/.termux-build/pypy3/src/pypy/objspace/std/objspace.py", line 114, in initialize
    self.make_builtins()
   File "/home/builder/.termux-build/pypy3/src/pypy/interpreter/baseobjspace.py", line 665, in make_builtins
    self.install_mixedmodule(mixedname)
   File "/home/builder/.termux-build/pypy3/src/pypy/interpreter/baseobjspace.py", line 706, in install_mixedmodule
    self.setbuiltinmodule(mixedname)
   File "/home/builder/.termux-build/pypy3/src/pypy/interpreter/baseobjspace.py", line 543, in setbuiltinmodule
    mod = Module(self, self.newtext(name))
   File "/home/builder/.termux-build/pypy3/src/pypy/module/pyexpat/moduledef.py", line 70, in __init__
    ver = space.unwrap(interp_pyexpat.get_expat_version(space))
   File "/home/builder/.termux-build/pypy3/src/pypy/module/pyexpat/interp_pyexpat.py", line 411, in get_expat_version
    return space.newtext(rffi.constcharp2str(XML_ExpatVersion()))
   File "/home/builder/.termux-build/pypy3/src/rpython/rtyper/lltypesystem/rffi.py", line 321, in wrapper
    res = call_external_function(*real_args)
   File "<2682-codegen /home/builder/.termux-build/pypy3/src/rpython/rtyper/lltypesystem/rffi.py:217>", line 13, in call_external_function
    res = funcptr()
   File "/home/builder/.termux-build/pypy3/src/rpython/rtyper/lltypesystem/lltype.py", line 1384, in __call__
    return callb(*args)
   File "/home/builder/.termux-build/pypy3/src/rpython/rtyper/lltypesystem/ll2ctypes.py", line 1336, in __call__
    self.natural_arity)
   File "/home/builder/.termux-build/pypy3/src/rpython/rtyper/lltypesystem/ll2ctypes.py", line 1265, in get_ctypes_callable
    clib = dllclass._dlltype(libpath, **load_library_kwargs)
   File "/home/builder/.termux-build/pypy3/src/pypy2.7-v7.3.19-linux64/lib-python/2.7/ctypes/__init__.py", line 369, in __init__
    pypy_dll = _ffi.CDLL(name, mode, handle)
[translation:ERROR] OSError: Cannot load library /home/builder/.termux-build/pypy3/src/usession-dir/usession-release-pypy3.11-v7.3.19-0/shared_cache/externmod.so: /home/builder/.termux-build/pypy3/src/usession-dir/usession-release-pypy3.11-v7.3.19-0/shared_cache/externmod.so: cannot open shared object file: No such file or directory

The only way I can figure out how to bypass this error in building pypy3 is by using the bionic-libc python2 to bootstrap.

- Large amount of changes, see GitHub PR description for details - Also touches `python2`

licy183 · 2025-04-28T07:10:06Z

Actually I don't think that I'll accept this because it will take more build time, and on-device build for pypy is not really necessary because of the insufficient RAM. This PR will run native translation (from proot-based bionic-aarch64 rootfs), not cross translation (from proot-based bionic-x86_64 to proot-based bionic-aarch64), and takes longer build time. I'm not sure whether it will finish the build within 6 hours.

robertkirkman · 2025-04-28T07:17:14Z

Actually I don't think that I'll accept this because it will take more build time, and on-device build for pypy is not really necessary because of the insufficient RAM. This PR will run native translation (from proot-based bionic-aarch64 rootfs), not cross translation (from proot-based bionic-x86_64 to proot-based bionic-aarch64), and takes longer build time. I'm not sure whether it will finish the build within 6 hours.

The build will finish within 6 hours (I tested first in my own GitHub Actions), but I understand exactly why it will not be accepted.

The biggest concern I have is, that I would like to deblobbify (remove propagated binaries that do not have an absolutely verifiable compilation origin) the build.sh of pypy3, but I don't know how to do that and still keep the cross translation.

I think the true best way to do that would be to have a GNU/Linux PyPy2 used for the translation-step (like packages/pypy/build.sh uses after the commit linked in the description), but I could not figure out how to do it successfully.

Do you think you might know how to successfully deblobbify the build.sh of pypy3 but keep the build time optimization?

robertkirkman · 2025-04-28T09:04:59Z

Indeed, for another case that is not fixed yet, I do understand that the current termux-proot-run's QEMU and proot do not come from GitHub Actions or a popular distro's package,

termux-packages/scripts/build/setup/termux_setup_proot.sh

Lines 21 to 31 in 4756daa

    
           termux_download https://github.com/proot-me/proot/releases/download/v"$TERMUX_PROOT_VERSION"/proot-v"$TERMUX_PROOT_VERSION"-x86_64-static \ 
        
           	"$TERMUX_PROOT_BIN/proot" \ 
        
           	d1eb20cb201e6df08d707023efb000623ff7c10d6574839d7bb42d0adba6b4da 
        
           chmod +x "$TERMUX_PROOT_BIN"/proot 
        
           declare -A checksums=( 
        
           	["aarch64"]="dce64b2dc6b005485c7aa735a7ea39cb0006bf7e5badc28b324b2cd0c73d883f" 
        
           	["arm"]="9f07762a3cd0f8a199cb5471a92402a4765f8e2fcb7fe91a87ee75da9616a806" 
        
           ) 
        
           if [[ "$TERMUX_ARCH" == "aarch64" ]] || [[ "$TERMUX_ARCH" == "arm" ]]; then 
        
           	termux_download https://github.com/multiarch/qemu-user-static/releases/download/v"$TERMUX_QEMU_VERSION"/qemu-"${TERMUX_ARCH/i686/i386}"-static \

but I believe that will be easier to fix, because there are many other ways to get them that can replace them.

For example, qemu-user-static can be replaced simply with the /usr/bin/qemu-aarch64-static that is already in the docker image
For example, proot is formed from not too much code and it can be compiled from source for the GNU/Linux host somewhat quickly if just libtalloc-dev and libarchive-dev are downloaded and used somehow.

I am trying to focus on one step at a time rather than changing absolutely everything all at once.

licy183 · 2025-04-28T13:50:47Z

I really don't think it is necessary to compile proot from source. The binary is downloaded from their maintainer's GitHub Release. There are many prebuilt binaries in our building system which is downloaded from GitHub Release or some other sources.

Using system-provided qemu-system-static is also not a good solution. Although Ubuntu 24.04's qemu is OK, it used to be not working on Ubuntu 20.04's. clang hangs out for unknown reason and it is fixed since qemu 6.2 which is landed in Ubuntu 22.04 and above.

IIRC, on QEMU < 5, python crashes (

termux-packages/packages/pypy/build.sh

Line 84 in 68a94d5

# Check qemu version, must greater than 6.0.0, since qemu 4/5 cannot run python

); on QEMU <= 6, clang++ hangs for some reason (termux/termux-docker#28,

termux-packages/packages/pypy/build.sh

Line 188 in 68a94d5

# FIXME: because clang under qemu sometimes hangs.

); and on QEMU < 7.1, python2 (2.7.18) crashes with OpenSSL (

termux-packages/packages/pypy/build.sh

Line 182 in 68a94d5

    
           # `pip2` is set up in the postinst script of `python2`, but it will segfault on aarch64. Install it manually.

).

I've tried to do cross translation for pypy3 from x86_64-linux-gnu to *-linux-android, but I haven't made it so far. It crashes on 32-bit arches. See https://github.com/termux/termux-packages/tree/dev/pypy3-cross-translation

robertkirkman · 2025-04-28T22:11:47Z

I really don't think it is necessary to compile proot from source. The binary is downloaded from their maintainer's GitHub Release. There are many prebuilt binaries in our building system which is downloaded from GitHub Release or some other sources.

Ok I understand. I would prefer using precompiled binaries only when they come from origins that have more than a certain level of popularity, but this level is different for every different person. You consider proot a popular enough software that using the precompiled release is acceptable for you. However, there is also an additional point which is that the upstream proot already has a release 5.4.0, but they did not release any precompiled binaries for it, so we are stuck with 5.3.0 until we set up our own compilation of it.

Using system-provided qemu-system-static is also not a good solution. Although Ubuntu 24.04's qemu is OK, it used to be not working on Ubuntu 20.04's.

If the same logic you are using is applied to all softwares (not only QEMU), then the assumption is that everything that is currently in the setup-ubuntu.sh script should ideally be removed as much as possible and replaced with alternative downloads or compilations that have a version independent from the Ubuntu version. There are many, but one example:

termux-packages/scripts/setup-ubuntu.sh

Line 216 in 72c1bbc

PACKAGES+=" gobject-introspection"

termux-packages/packages/glib/meson.build.patch

Line 95 in b5b29e9

+# TODO: Remove this after gobject-introspection 1.84.0 update in host system

the version of Ubuntu gobject-introspection in setup-ubuntu.sh already affects the builds of Termux packages because of its version being a specific version, similar to the concept you are explaining about QEMU (but for other packages that are not QEMU).

I've tried to do cross translation for pypy3 from x86_64-linux-gnu to *-linux-android, but I haven't made it so far. It crashes on 32-bit arches. See https://github.com/termux/termux-packages/tree/dev/pypy3-cross-translation

Thank you very much I did not know you made that already!!
That is farther than I was able to get, for me my attempt did not succeed on any architecture.
I will try to help. If that version already works for 64-bit, then maybe I can fix it to also work for 32-bit.

licy183 · 2025-04-28T22:37:12Z

the version of Ubuntu gobject-introspection in setup-ubuntu.sh already affects the builds of Termux packages because of its version being a specific version

There are some differences between gobject-introspection and qemu-user-static. It is quite difficult to use gobject-introspection of specific version because it has too many dependencies, but it is easy to use qemu-user-static of specific version because it is compiled statically.

bump(main/pypy3): 7.3.19

5ce8098

- Large amount of changes, see GitHub PR description for details - Also touches `python2`

robertkirkman requested review from licy183, Grimler91 and TomJo2000 as code owners April 28, 2025 06:38

robertkirkman marked this pull request as draft April 28, 2025 22:18

robertkirkman mentioned this pull request Jun 24, 2025

[Bug]: Packages Failed To Build #24930

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

bump(main/pypy3): 7.3.19 #24525

bump(main/pypy3): 7.3.19 #24525

robertkirkman commented Apr 28, 2025

Uh oh!

licy183 commented Apr 28, 2025

Uh oh!

robertkirkman commented Apr 28, 2025

Uh oh!

robertkirkman commented Apr 28, 2025

Uh oh!

licy183 commented Apr 28, 2025 •

edited

Loading

Uh oh!

robertkirkman commented Apr 28, 2025

Uh oh!

licy183 commented Apr 28, 2025

Uh oh!

Uh oh!

Uh oh!

bump(main/pypy3): 7.3.19 #24525

Are you sure you want to change the base?

bump(main/pypy3): 7.3.19 #24525

Conversation

robertkirkman commented Apr 28, 2025

downside: build-time performance regression

Uh oh!

licy183 commented Apr 28, 2025

Uh oh!

robertkirkman commented Apr 28, 2025

Uh oh!

robertkirkman commented Apr 28, 2025

Uh oh!

licy183 commented Apr 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

robertkirkman commented Apr 28, 2025

Uh oh!

licy183 commented Apr 28, 2025

Uh oh!

Uh oh!

licy183 commented Apr 28, 2025 •

edited

Loading