tree-wide: port debpython to termux #23652

robertkirkman · 2025-03-07T01:55:39Z

Partial fix (one part of the fix for) [Bug]: mesa & mesa-vulkan-icd-freedreno-dri3 build error #21658

What is debpython?

debpython is the commands py3compile and py3clean from Debian. I
am calling them that because a large chunk of their source code is
found inside a folder inside Debian's source code named "debpython"
- https://salsa.debian.org/cpython-team/python3-defaults/-/tree/5348f704668c0b6c360b6c6fb10153b9c2898af5
rather than packaging .pyc files into .deb files, py3compile and
py3clean can be called from postinst and prerm scripts to
generate all .pyc for the .py files in the package immediately
after the package is installed, and remove all .pyc files immediately
before uninstalling the package, respectively
fixes the error trying to overwrite '/data/data/com.termux/files/usr/lib/python3.12/__pycache__/cProfile.cpython-312.pyc' when packages were built on-device, but at the same time, also:
prevents the warnings dpkg: warning: while removing python, directory '/data/data/com.termux/files/usr/lib/python3.12/site-packages' not empty so not removed, as long as no packages were installed using pip.
Not implemented for pypy, pypy3, or python2 packages, only python3 packages currently.

How to add a new Python package after this?

Everything is the same, except, now, this block is no longer necessary in build.sh.

termux_step_create_debscripts() {
	cat <<- EOF > ./postinst
	#!$TERMUX_PREFIX/bin/sh
	echo "Installing dependencies through pip..."
	pip3 install ${TERMUX_PKG_PYTHON_TARGET_DEPS//, / }
	EOF
}

Instead, scripts/build/termux_step_create_python_debscripts.sh can now detect the presence of pip package lists in $TERMUX_PKG_PYTHON_TARGET_DEPS, $TERMUX_SUBPKG_PYTHON_TARGET_DEPS, and the METADATA file of the Python package if it exists, and automatically insert them as a block into the postinst script for all relevant packages.
$TERMUX_PKG_PYTHON_TARGET_DEPS is used for pip dependencies that are both on-device build-time and on-device run-time dependencies, and $TERMUX_PKG_PYTHON_RUNTIME_DEPS is used for runtime-only pip dependencies. $TERMUX_PKG_PYTHON_RUNTIME_DEPS overrides $TERMUX_PKG_PYTHON_TARGET_DEPS for runtime dependencies,
- i.e. if TERMUX_PKG_PYTHON_RUNTIME_DEPS is not specified, but TERMUX_PKG_PYTHON_TARGET_DEPS is, then TERMUX_PKG_PYTHON_TARGET_DEPS will be used as both on-device build and on-device runtime dependencies,
- but if TERMUX_PKG_PYTHON_RUNTIME_DEPS is specified, then TERMUX_PKG_PYTHON_TARGET_DEPS, if specified, is used only for on-device build-time dependencies.
python-pip is also automatically inserted into the runtime dependencies of every package that needs it, appropriately.

licy183 · 2025-03-07T18:12:30Z

This folder should only be excluded only when performing on-device build. In some packages, like python, this content in this folder is managed by apt. It is quite annoying to see some warnings like xxx directory is not empty.

robertkirkman · 2025-03-08T04:10:56Z

This folder should only be excluded only when performing on-device build. In some packages, like python, this content in this folder is managed by apt

Should the content in these folders remain managed by apt in any case?

In Debian, I believe these folders are not managed by apt.

https://packages.debian.org/sid/amd64/libpython3.12-minimal/filelist

In Arch Linux, I believe these folders are managed by pacman.

https://archlinux.org/packages/core/x86_64/python/ (file list)

Fedora also ships them

https://packages.fedoraproject.org/pkgs/python3.12/python3-libs/fedora-40.html

it seems to me like it is optional whether to include them in the package, because I think Python can automatically generate them, but this problem is very difficult because, I think Arch Linux and Fedora might ship them for performance.

At least this does not break Python, I think, because a python package built from this change and installed in Termux after rm -rf $PREFIX/lib/python3.12 still works and runs.

It is quite annoying to see some warnings like xxx directory is not empty.

I do not think there is any way to get rid of this warning, because on Termux python-pip generates and stores the __pycache__ folders into the site-packages folder in the same $PREFIX/lib/python3.12/ folder where the $PREFIX/lib/python3.12/zoneinfo/_zoneinfo.py files are, etc.

Example: warning always reproducible with current packages

pkg install python-pip
pip install setuptools
pkg remove python

Removing python-pip (25.0.1) ...
Removing python-ensurepip-wheels (3.12.9) ...
Removing python (3.12.9) ...
dpkg: warning: while removing python, directory '/data/data/com.termux/files/usr/lib/python3.12/site-packages' not empty so not removed
Processing triggers for man (1.14.6-1) ...

That happens without this change.

Can you think of any way to:

prevent the __pycache__ folders from being incorrectly included in non-Python packages during on-device build-package.sh, to fix trying to overwrite '/data/data/com.termux/files/usr/lib/python3.12/__pycache__/cProfile.cpython-312.pyc', which is also in package python 3.12.9
but still make sure __pycache__ folders are included in the package only when they actually belong to the package currently being built, so they can be shipped for a Python startup performance boost?

I do not actually notice the performance difference at the startup of Python, when I try to test for it, but I believe someone else might.

licy183 · 2025-03-08T07:36:09Z

In Debain, they have a postinst script to compile the modules when new packages are installed, and a prerm script to clean up __pycache__ dir and *.pyc files. I have done the same cleanup logic in pypy3 on Termux.

See /usr/share/python3/runtime.d/public_modules.rt{install,remove} in Debain.

robertkirkman · 2025-03-14T08:24:22Z

In Debain, they have a postinst script to compile the modules when new packages are installed, and a prerm script to clean up __pycache__ dir and *.pyc files. I have done the same cleanup logic in pypy3 on Termux.

See /usr/share/python3/runtime.d/public_modules.rt{install,remove} in Debain.

ok I did it, is this what you meant?

robertkirkman · 2025-03-15T10:53:32Z

@Maxython
Ok, to continue the discussion here,

now I have:

unified almost all python debscript creation into a single function, removing the duplicated code involving $TERMUX_PKG_PYTHON_TARGET_DEPS from many packages and also making it not necessary to explicitly call termux_step_create_python_debscripts() from any build.sh anymore
tried to make this debpython compatible with pacman by making it try to use the pacman -Qql command if the dpkg command is not found

however, I am not sure whether the pacman -Qql command is completely compatible to the point that it will work in all cases without problems. pacman -Qql does print equivalent data in very similar formatting to dpkg -L, but it is not 100% identical formatting to dpkg -L, so I am not sure whether it will work perfectly without a lot of testing.

TomJo2000

Thanks for putting in the effort.
Looks like I accidentally created a merge conflict for you with one of my earlier PRs, sorry about that.

robertkirkman · 2025-03-16T03:05:50Z

That's ok,
something I have noticed is that, licy183 told me somewhere else that TERMUX_PKG_REVISION should not be incremented for PRs that are excluded from CI. The reason I excluded the PR from CI is because I previously learned here that if too many packages are built in a single PR, sometimes the behavior will diverge from the behavior of the same large build command run in the docker builder locally (even when packages are not polluting the builds of other packages)

However, the reason why many packages are marked with revision-bumps in this is because, that is marking all packages that a test I did indicates will have changed control.tar.xz contents if they are rebuilt after this PR is merged.

Because of that, I think probably what I need to do is split this into several PRs one after another that do not have CI disabled.

Maxython · 2025-07-18T13:13:19Z

If you have a specific idea for how to fix the problem of Errno 2] No such file or directory: '/data/data/com.termux/files/usr/lib/python3.12/site-packages/nala/cache.py' in a better way, then feel free to force push this branch with any change you want to show, and we can see how that way looks

The idea for solving the problem: if in the list of modules to install during the installation of packages there is a module whose name matches the name of the package, then the --upgrade flag will be disabled.

scripts/build/termux_step_create_python_debscripts.sh

robertkirkman · 2025-07-18T20:33:06Z

Ok, I have tested installing the current version on two termux-apt devices with two different CPU architectures, and one clean installed and one not clean installed, and I can confirm there are no more fatal errors anywhere in this PR.

However, the one remaining possible problem I see is, I still frequently see these errors:

licy183 mentioned these errors first, but unfortunately, I don't know how to fix these.

They are non-fatal errors, because after they show, "Successfully installed" prints anyway.

Also, they do not necessarily produce runtime errors in the python programs, because for example, after this I tested commands like manim --show-splash init project, and they appear to work without errors.

On the other hand, I am not sure whether these errors indicate that subtle, hard-to-find crashes in some programs might appear.

Do you think it is acceptable to allow these errors, or do you know of any way to prevent them?

robertkirkman · 2025-07-18T20:41:55Z

I checked and noticed that the dependencies logged as "not installed", do actually appear as installed at the end after all debscripts finish running.

for example, mapbox-earcut 1.0.2 is installed after the apt install command finishes, but I didn't manually install it.

This suggests the error might be spurious, or also, it could be a coincidence, and maybe mapbox-earcut was installed by a different debscript somehow, from the one that printed that message. I'm not sure which is the case.

licy183 · 2025-07-19T04:26:40Z

Do you think it is acceptable to allow these errors, or do you know of any way to prevent them?

As I commented before, if you install some package manually (pip install manim), pip will automatically change these dependencies to correct version.

robertkirkman · 2025-07-19T07:38:35Z

Thank you for explaining, I am sorry for being dense about this detail. I think I am beginning to get an idea of what should be done. I will think about this and make a solution.

robertkirkman · 2025-07-25T08:35:10Z

After the most recent changes, when I test fully recompiling and reinstalling every single package here at once into a clean Termux installation, the amount of visible instances of ERROR: pip's dependency resolver does not currently take into account all the packages that are installed has not been 100% completely eliminated, but they have been greatly reduced, and all of the instances remaining seem to occur because apt installs all the python package files into $PREFIX before ever running any debscripts, then runs the postinst scripts one by one for each package, so these messages sometimes print temporarily while not all packages have had their postinst scripts run yet - but I believe that all PyPi dependencies are now installed correctly by the time all postinst scripts have finished running.

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
borgbackup 1.4.1 requires msgpack<=1.1.0,>=1.0.3, which is not installed.
matplotlib 3.10.3 requires cycler>=0.10, which is not installed.
matplotlib 3.10.3 requires fonttools>=4.22.0, which is not installed.
matplotlib 3.10.3 requires kiwisolver>=1.3.1, which is not installed.
matplotlib 3.10.3 requires pyparsing>=2.3.1, which is not installed.
matplotlib 3.10.3 requires python-dateutil>=2.7, which is not installed.
torch 2.6.0 requires filelock, which is not installed.
torch 2.6.0 requires fsspec, which is not installed.
torch 2.6.0 requires jinja2, which is not installed.
torch 2.6.0 requires networkx, which is not installed.
torch 2.6.0 requires setuptools; python_version >= "3.12", which is not installed.
torch 2.6.0 requires typing-extensions>=4.10.0, which is not installed.
torch 2.6.0 requires sympy==1.13.1; python_version >= "3.9", but you have sympy 1.14.0 which is incompatible.
Successfully installed coloredlogs-15.0.1 flatbuffers-25.2.10 humanfriendly-10.0 mpmath-1.3.0 packaging-25.0 protobuf-6.31.1 sympy-1.14.0
Setting up ffmpeg (7.1.1-4) ...
Setting up python-torch (2.6.0-1) ...
Installing dependencies for python-torch through pip...
Requirement already satisfied: torch in /data/data/com.termux/files/usr/lib/python3.12/site-packages (2.6.0)
Collecting filelock (from torch)
  Downloading filelock-3.18.0-py3-none-any.whl.metadata (2.9 kB)
Collecting typing-extensions>=4.10.0 (from torch)
  Downloading typing_extensions-4.14.1-py3-none-any.whl.metadata (3.0 kB)
Collecting setuptools (from torch)
  Downloading setuptools-80.9.0-py3-none-any.whl.metadata (6.6 kB)
Collecting sympy==1.13.1 (from torch)
  Downloading sympy-1.13.1-py3-none-any.whl.metadata (12 kB)
Collecting networkx (from torch)
  Downloading networkx-3.5-py3-none-any.whl.metadata (6.3 kB)
Collecting jinja2 (from torch)
  Downloading jinja2-3.1.6-py3-none-any.whl.metadata (2.9 kB)
Collecting fsspec (from torch)
  Downloading fsspec-2025.7.0-py3-none-any.whl.metadata (12 kB)
Requirement already satisfied: mpmath<1.4,>=1.1.0 in /data/data/com.termux/files/usr/lib/python3.12/site-packages (from sympy==1.13.1->torch) (1.3.0)

In particular, I noticed by looking closely that I think all of the dependencies are now being resolved as well as possible for every single package, one by one as the postinst scripts for each package explicitly install the pip-facing package name without --upgrade:

torch 2.6.0 requires filelock, which is not installed.

...

Setting up python-torch (2.6.0-1) ...
Installing dependencies for python-torch through pip...
Requirement already satisfied: torch in /data/data/com.termux/files/usr/lib/python3.12/site-packages (2.6.0)
Collecting filelock (from torch)
  Downloading filelock-3.18.0-py3-none-any.whl.metadata (2.9 kB)

@Maxython The amount of changes I have made to do this is quite large, and I am not sure whether you agree with all of the changes. Please let me know if you think everything I have done here is a good idea.

however, after you have finished reviewing my new changes, and any follow-up changes have finished, I believe this PR will be ready.

robertkirkman

EDIT: this comment is an accident, please ignore this comment and read the other one below.

          |
          v

robertkirkman · 2025-07-25T10:32:04Z

scripts/buildorder.py

@@ -39,7 +39,7 @@ def parse_build_file_dependencies_with_vars(path, vars):
    with open(path, encoding="utf-8") as build_script:
        for line in build_script:
            if line.startswith(vars):
-                dependencies_string = line.split('DEPENDS=')[1]
+                dependencies_string = line.split('DEPS=' if 'DEPS=' in line else 'DEPENDS=')[1]


@Maxython I am a little bit confused about this line, and I don't know if I need to change it to match my other recent changes somehow. If it's easy for you to explain, could you explain a little bit about what this line is supposed to do?

- debpython is the commands `py3compile` and `py3clean` from Debian. I am calling them that because a large chunk of their source code is found inside a folder inside Debian's source code named "debpython" - https://salsa.debian.org/cpython-team/python3-defaults/-/tree/5348f704668c0b6c360b6c6fb10153b9c2898af5 - rather than packaging `.pyc` files into packages, `py3compile` and `py3clean` can be called from `postinst` and `prerm` scripts to generate all `.pyc` for the `.py` files in the package immediately after the package is installed, and remove all `.pyc` files immediately before uninstalling the package, respectively - fixes the error `trying to overwrite '/data/data/com.termux/files/usr/lib/python3.12/__pycache__/cProfile.cpython-312.pyc'` when packages were built on-device, but at the same time, also: - prevents the warnings `dpkg: warning: while removing python, directory '/data/data/com.termux/files/usr/lib/python3.12/site-packages' not empty so not removed` as long as no packages were installed using `pip` - The revision-bumps that change `control.tar.xz` (debscripts) of packages only and are not otherwise visible are omitted from this PR, and will come in a second PR afterward. - The `termux_step_create_python_debscripts.sh` can configure work on debpython (i.e. its `py3copile` and `py3clean` commands) from the glibc package `python-glibc`, if some glibc package is being compiled. - New variables have been implemented: - `TERMUX_PYTHON_CROSSENV_BUILDHOME` - location of crossenv's python build libraries. - `TERMUX_PKG_PYTHON_RUNTIME_DEPS` - configures the installation of the python modules via pip3 in the pkg's debscripts. If not configured in the package, it will use the value from `TERMUX_PKG_PYTHON_TARGET_DEPS`. If the variable is set to `false`, then the customization of installing python modules will be disabled, even if the `TERMUX_PKG_PYTHON_TARGET_DEPS` variable is set in the package. - `TERMUX_SUBPKG_PYTHON_RUNTIME_DEPS` - configures the installation of the python modules via pip3 in the subpkg's debscripts. - Implemented reconfiguration of prefixes in python module `sysconfig` and setting in `TERMUX_PYTHON_CROSSENV_BUILDHOME`, so that python modules from crossenv building can specify system paths of termux for correct compilation. - Added automatic addition of `python-glibc{-glibc}` dependency when using the `TERMUX_PKG_PYTHON_RUNTIME_DEPS` (for pkg; will be disabled, i.e. will not be added, if the variable is set to `false`) or `TERMUX_SUBPKG_PYTHON_RUNTIME_DEPS` (for subpkg) value. - setup-termux: - add comment and disabled dependency 'pandoc' to prepare for steps towards fixing on-device build of 'nala' - add 'python-pip' and implementation of installing 'system-wide' (Termux-wide) pip packages outside of venv in order to work towards closer accuracy of setup-termux.sh to setup-ubuntu.sh - add 'itstool' from PyPi system-wide to approximate the installation through 'apt' of 'itstool' system-wide in the Ubuntu cross-builder image Co-authored-by: Maxython <mixython@gmail.com>

@robertkirkman

- packages/borgbackup - packages/electrum - packages/hash-slinger - packages/lv2 - packages/manim - packages/nala - packages/python-cryptography - packages/python-tldp - packages/python-xlib - packages/rdiff-backup - packages/rdircd - packages/sabnzbd - packages/srt2vobsub - packages/termux-gui-package - packages/termux-gui-pm - packages/ytui-music - x11-packages/orca - x11-packages/pyqt5 - packages/matplotlib - packages/python-onnxruntime - packages/python-torch - packages/python-torchvision - packages/python-yt-dlp - x11-packages/xfce4-panel-profiles - disabled-packages/python-pandas - packages/calcurse - packages/units - root-packages/frida - x11-packages/inkscape Co-authored-by: @robertkirkman

robertkirkman requested a review from Grimler91 as a code owner March 7, 2025 01:55

robertkirkman marked this pull request as draft March 8, 2025 00:25

robertkirkman mentioned this pull request Mar 12, 2025

scripts: fix conflicts due to __pycache__ #23736

Open

robertkirkman force-pushed the exclude-pycache-during-extract-into-massagedir branch from eef60d2 to 7456b5b Compare March 14, 2025 08:17

robertkirkman changed the title ~~fix(scripts/build/termux_step_extract_into_massagedir): exclude __pycache__ directories~~ tree-wide: port debpython to termux Mar 14, 2025

robertkirkman marked this pull request as ready for review March 14, 2025 08:22

robertkirkman requested review from TomJo2000 and finagolfin as code owners March 14, 2025 08:22

robertkirkman force-pushed the exclude-pycache-during-extract-into-massagedir branch 5 times, most recently from 84e2f31 to 5d3a7a9 Compare March 15, 2025 10:44

TomJo2000 approved these changes Mar 16, 2025

View reviewed changes

robertkirkman marked this pull request as draft March 16, 2025 02:54

robertkirkman force-pushed the exclude-pycache-during-extract-into-massagedir branch from 5d3a7a9 to cd049e3 Compare March 16, 2025 16:07

robertkirkman marked this pull request as ready for review March 16, 2025 17:47

Maxython force-pushed the exclude-pycache-during-extract-into-massagedir branch from 9c780b7 to e7253ee Compare March 17, 2025 12:17

robertkirkman marked this pull request as draft March 17, 2025 19:19

Maxython force-pushed the exclude-pycache-during-extract-into-massagedir branch 3 times, most recently from 5b68174 to 9266459 Compare March 18, 2025 10:12

Maxython force-pushed the exclude-pycache-during-extract-into-massagedir branch 2 times, most recently from f1ec5dd to 8331f6f Compare April 1, 2025 08:13

Maxython force-pushed the exclude-pycache-during-extract-into-massagedir branch from e966bc9 to d53b6cf Compare July 18, 2025 13:09

robertkirkman commented Jul 18, 2025

View reviewed changes

scripts/build/termux_step_create_python_debscripts.sh Outdated Show resolved Hide resolved

robertkirkman marked this pull request as draft July 18, 2025 16:58

robertkirkman marked this pull request as ready for review July 18, 2025 20:21

robertkirkman mentioned this pull request Jul 18, 2025

enhance(main/libopenmpt, mpg123, chromaprint): split libopenmpt, mpg123 and chromaprint executables from their libraries #25227

Open

robertkirkman marked this pull request as draft July 19, 2025 07:38

robertkirkman force-pushed the exclude-pycache-during-extract-into-massagedir branch from d53b6cf to b7df09e Compare July 25, 2025 08:33

robertkirkman marked this pull request as ready for review July 25, 2025 08:33

robertkirkman force-pushed the exclude-pycache-during-extract-into-massagedir branch 2 times, most recently from 0cdccc1 to f313757 Compare July 25, 2025 10:22

robertkirkman commented Jul 25, 2025

View reviewed changes

robertkirkman force-pushed the exclude-pycache-during-extract-into-massagedir branch 6 times, most recently from 87abddc to ebfbea5 Compare July 25, 2025 20:13

robertkirkman mentioned this pull request Jul 25, 2025

fix(x11/dconf-editor): Use download.gnome.org source URL like other GNOME packages #25443

Merged

robertkirkman force-pushed the exclude-pycache-during-extract-into-massagedir branch 2 times, most recently from 1e8062c to 4a8e5ae Compare July 26, 2025 05:57

robertkirkman and others added 3 commits July 27, 2025 06:43

robertkirkman debpython temporary commit 3

aa0b286

robertkirkman force-pushed the exclude-pycache-during-extract-into-massagedir branch from 4a8e5ae to aa0b286 Compare July 27, 2025 11:45

Uh oh!

tree-wide: port debpython to termux #23652

Are you sure you want to change the base?

tree-wide: port debpython to termux #23652

Uh oh!

Conversation

robertkirkman commented Mar 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is debpython?

How to add a new Python package after this?

Uh oh!

licy183 commented Mar 7, 2025

Uh oh!

robertkirkman commented Mar 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

licy183 commented Mar 8, 2025

Uh oh!

robertkirkman commented Mar 14, 2025

Uh oh!

robertkirkman commented Mar 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TomJo2000 left a comment

Choose a reason for hiding this comment

Uh oh!

robertkirkman commented Mar 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Maxython commented Jul 18, 2025

Uh oh!

Uh oh!

robertkirkman commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

robertkirkman commented Jul 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

licy183 commented Jul 19, 2025

Uh oh!

robertkirkman commented Jul 19, 2025

Uh oh!

robertkirkman commented Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

robertkirkman left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

robertkirkman Jul 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

robertkirkman commented Mar 7, 2025 •

edited

Loading

robertkirkman commented Mar 8, 2025 •

edited

Loading

robertkirkman commented Mar 15, 2025 •

edited

Loading

robertkirkman commented Mar 16, 2025 •

edited

Loading

robertkirkman commented Jul 18, 2025 •

edited

Loading

robertkirkman commented Jul 18, 2025 •

edited

Loading

robertkirkman commented Jul 25, 2025 •

edited

Loading

robertkirkman left a comment •

edited

Loading