tree-wide: port debpython to termux #23652

robertkirkman · 2025-03-07T01:55:39Z

What is debpython?

debpython is the commands py3compile and py3clean from Debian. I
am calling them that because a large chunk of their source code is
found inside a folder inside Debian's source code named "debpython"
- https://salsa.debian.org/cpython-team/python3-defaults/-/tree/5348f704668c0b6c360b6c6fb10153b9c2898af5
rather than packaging .pyc files into .deb files, py3compile and
py3clean can be called from postinst and prerm scripts to
generate all .pyc for the .py files in the package immediately
after the package is installed, and remove all .pyc files immediately
before uninstalling the package, respectively
fixes the error trying to overwrite '/data/data/com.termux/files/usr/lib/python3.12/__pycache__/cProfile.cpython-312.pyc' when packages were built on-device, but at the same time, also:
prevents the warnings dpkg: warning: while removing python, directory '/data/data/com.termux/files/usr/lib/python3.12/site-packages' not empty so not removed, as long as no packages were installed using pip.
Not implemented for pypy, pypy3, or python2 packages, only python3 packages currently.

How to add a new Python package after this?

Everything is the same, except, now, this block is no longer necessary in build.sh.

termux_step_create_debscripts() {
	cat <<- EOF > ./postinst
	#!$TERMUX_PREFIX/bin/sh
	echo "Installing dependencies through pip..."
	pip3 install ${TERMUX_PKG_PYTHON_TARGET_DEPS//, / }
	EOF
}

Instead, scripts/build/termux_step_create_python_debscripts.sh can now detect the presence of pip package lists in $TERMUX_PKG_PYTHON_TARGET_DEPS, $TERMUX_SUBPKG_PYTHON_TARGET_DEPS, and the METADATA file of the Python package if it exists, and automatically insert them as a block into the postinst script for all relevant packages.
$TERMUX_PKG_PYTHON_TARGET_DEPS is used for pip dependencies that are both on-device build-time and on-device run-time dependencies, and $TERMUX_PKG_PYTHON_RUNTIME_DEPS is used for runtime-only pip dependencies. $TERMUX_PKG_PYTHON_RUNTIME_DEPS overrides $TERMUX_PKG_PYTHON_TARGET_DEPS for runtime dependencies,
- i.e. if TERMUX_PKG_PYTHON_RUNTIME_DEPS is not specified, but TERMUX_PKG_PYTHON_TARGET_DEPS is, then TERMUX_PKG_PYTHON_TARGET_DEPS will be used as both on-device build and on-device runtime dependencies,
- but if TERMUX_PKG_PYTHON_RUNTIME_DEPS is specified, then TERMUX_PKG_PYTHON_TARGET_DEPS, if specified, is used only for on-device build-time dependencies.
python-pip is also automatically inserted into the runtime dependencies of every package that needs it, appropriately.

licy183 · 2025-03-07T18:12:30Z

This folder should only be excluded only when performing on-device build. In some packages, like python, this content in this folder is managed by apt. It is quite annoying to see some warnings like xxx directory is not empty.

robertkirkman · 2025-03-08T04:10:56Z

This folder should only be excluded only when performing on-device build. In some packages, like python, this content in this folder is managed by apt

Should the content in these folders remain managed by apt in any case?

In Debian, I believe these folders are not managed by apt.

https://packages.debian.org/sid/amd64/libpython3.12-minimal/filelist

In Arch Linux, I believe these folders are managed by pacman.

https://archlinux.org/packages/core/x86_64/python/ (file list)

Fedora also ships them

https://packages.fedoraproject.org/pkgs/python3.12/python3-libs/fedora-40.html

it seems to me like it is optional whether to include them in the package, because I think Python can automatically generate them, but this problem is very difficult because, I think Arch Linux and Fedora might ship them for performance.

At least this does not break Python, I think, because a python package built from this change and installed in Termux after rm -rf $PREFIX/lib/python3.12 still works and runs.

It is quite annoying to see some warnings like xxx directory is not empty.

I do not think there is any way to get rid of this warning, because on Termux python-pip generates and stores the __pycache__ folders into the site-packages folder in the same $PREFIX/lib/python3.12/ folder where the $PREFIX/lib/python3.12/zoneinfo/_zoneinfo.py files are, etc.

Example: warning always reproducible with current packages

pkg install python-pip
pip install setuptools
pkg remove python

Removing python-pip (25.0.1) ...
Removing python-ensurepip-wheels (3.12.9) ...
Removing python (3.12.9) ...
dpkg: warning: while removing python, directory '/data/data/com.termux/files/usr/lib/python3.12/site-packages' not empty so not removed
Processing triggers for man (1.14.6-1) ...

That happens without this change.

Can you think of any way to:

prevent the __pycache__ folders from being incorrectly included in non-Python packages during on-device build-package.sh, to fix trying to overwrite '/data/data/com.termux/files/usr/lib/python3.12/__pycache__/cProfile.cpython-312.pyc', which is also in package python 3.12.9
but still make sure __pycache__ folders are included in the package only when they actually belong to the package currently being built, so they can be shipped for a Python startup performance boost?

I do not actually notice the performance difference at the startup of Python, when I try to test for it, but I believe someone else might.

licy183 · 2025-03-08T07:36:09Z

In Debain, they have a postinst script to compile the modules when new packages are installed, and a prerm script to clean up __pycache__ dir and *.pyc files. I have done the same cleanup logic in pypy3 on Termux.

See /usr/share/python3/runtime.d/public_modules.rt{install,remove} in Debain.

robertkirkman · 2025-03-14T08:24:22Z

In Debain, they have a postinst script to compile the modules when new packages are installed, and a prerm script to clean up __pycache__ dir and *.pyc files. I have done the same cleanup logic in pypy3 on Termux.

See /usr/share/python3/runtime.d/public_modules.rt{install,remove} in Debain.

ok I did it, is this what you meant?

robertkirkman · 2025-03-15T10:53:32Z

@Maxython
Ok, to continue the discussion here,

now I have:

unified almost all python debscript creation into a single function, removing the duplicated code involving $TERMUX_PKG_PYTHON_TARGET_DEPS from many packages and also making it not necessary to explicitly call termux_step_create_python_debscripts() from any build.sh anymore
tried to make this debpython compatible with pacman by making it try to use the pacman -Qql command if the dpkg command is not found

however, I am not sure whether the pacman -Qql command is completely compatible to the point that it will work in all cases without problems. pacman -Qql does print equivalent data in very similar formatting to dpkg -L, but it is not 100% identical formatting to dpkg -L, so I am not sure whether it will work perfectly without a lot of testing.

TomJo2000

Thanks for putting in the effort.
Looks like I accidentally created a merge conflict for you with one of my earlier PRs, sorry about that.

robertkirkman · 2025-03-16T03:05:50Z

That's ok,
something I have noticed is that, licy183 told me somewhere else that TERMUX_PKG_REVISION should not be incremented for PRs that are excluded from CI. The reason I excluded the PR from CI is because I previously learned here that if too many packages are built in a single PR, sometimes the behavior will diverge from the behavior of the same large build command run in the docker builder locally (even when packages are not polluting the builds of other packages)

However, the reason why many packages are marked with revision-bumps in this is because, that is marking all packages that a test I did indicates will have changed control.tar.xz contents if they are rebuilt after this PR is merged.

Because of that, I think probably what I need to do is split this into several PRs one after another that do not have CI disabled.

robertkirkman · 2025-10-18T22:33:35Z

@TomJo2000 Unfortunately, it seems like another problem related to

enhance(scripts/lint-packages.sh): make version check more robust. #26786

has occurred in this PR,

I rebased this PR past

bump(main/python): 3.12.12 #26930

just now, and this happened:

It seems to "think" that the current version is still Python 3.12.11-1, when it's definitely not. The current version is Python 3.12.12, and the diff shows that:

Do you know what might be going wrong here?

TomJo2000 · 2025-10-19T13:12:58Z

Looking into it.

robertkirkman · 2025-10-19T16:46:28Z

I force-pushed the branch again with no other changes today, and now it's passing the linter instead of failing, so the problem is intermittent. This is the run where it failed: https://github.com/termux/termux-packages/actions/runs/18621788032/job/53093893130 I will watch in the future for if it happens again.

…hronize `setup-termux.sh` closer to `setup-ubuntu.sh` - Follow-up to #24977 - Fixes #26373 - Split from #23652 because it does not really depend on the other things in that PR and it is now needed separately

- debpython is the commands `py3compile` and `py3clean` from Debian. I am calling them that because a large chunk of their source code is found inside a folder inside Debian's source code named "debpython" - https://salsa.debian.org/cpython-team/python3-defaults/-/tree/5348f704668c0b6c360b6c6fb10153b9c2898af5 - rather than packaging `.pyc` files into packages, `py3compile` and `py3clean` can be called from `postinst` and `prerm` scripts to generate all `.pyc` for the `.py` files in the package immediately after the package is installed, and remove all `.pyc` files immediately before uninstalling the package, respectively - fixes the error `trying to overwrite '/data/data/com.termux/files/usr/lib/python3.12/__pycache__/cProfile.cpython-312.pyc'` when packages were built on-device, but at the same time, also: - prevents the warnings `dpkg: warning: while removing python, directory '/data/data/com.termux/files/usr/lib/python3.12/site-packages' not empty so not removed` as long as no packages were installed using `pip` - The revision-bumps that change `control.tar.xz` (debscripts) of packages only and are not otherwise visible are omitted from this PR, and will come in a second PR afterward. - The `termux_step_create_python_debscripts.sh` can configure work on debpython (i.e. its `py3copile` and `py3clean` commands) from the glibc package `python-glibc`, if some glibc package is being compiled. - New variables have been implemented: - `TERMUX_PYTHON_CROSSENV_BUILDHOME` - location of crossenv's python build libraries. - `TERMUX_PKG_PYTHON_RUNTIME_DEPS` - configures the installation of the python modules via pip3 in the pkg's debscripts. If not configured in the package, it will use the value from `TERMUX_PKG_PYTHON_TARGET_DEPS`. If the variable is set to `false`, then the customization of installing python modules will be disabled, even if the `TERMUX_PKG_PYTHON_TARGET_DEPS` variable is set in the package. - `TERMUX_SUBPKG_PYTHON_RUNTIME_DEPS` - configures the installation of the python modules via pip3 in the subpkg's debscripts. - Implemented reconfiguration of prefixes in python module `sysconfig` and setting in `TERMUX_PYTHON_CROSSENV_BUILDHOME`, so that python modules from crossenv building can specify system paths of termux for correct compilation. - Added automatic addition of `python-glibc{-glibc}` dependency when using the `TERMUX_PKG_PYTHON_RUNTIME_DEPS` (for pkg; will be disabled, i.e. will not be added, if the variable is set to `false`) or `TERMUX_SUBPKG_PYTHON_RUNTIME_DEPS` (for subpkg) value. - setup-termux: - add comment and disabled dependency 'pandoc' to prepare for steps towards fixing on-device build of 'nala' - add 'python-pip' and implementation of installing 'system-wide' (Termux-wide) pip packages outside of venv in order to work towards closer accuracy of setup-termux.sh to setup-ubuntu.sh - add 'itstool' from PyPi system-wide to approximate the installation through 'apt' of 'itstool' system-wide in the Ubuntu cross-builder image Co-authored-by: Maxython <mixython@gmail.com>

robertkirkman requested a review from Grimler91 as a code owner March 7, 2025 01:55

robertkirkman marked this pull request as draft March 8, 2025 00:25

robertkirkman mentioned this pull request Mar 12, 2025

scripts: fix conflicts due to __pycache__ #23736

Open

robertkirkman force-pushed the exclude-pycache-during-extract-into-massagedir branch from eef60d2 to 7456b5b Compare March 14, 2025 08:17

robertkirkman changed the title ~~fix(scripts/build/termux_step_extract_into_massagedir): exclude __pycache__ directories~~ tree-wide: port debpython to termux Mar 14, 2025

robertkirkman marked this pull request as ready for review March 14, 2025 08:22

robertkirkman requested review from TomJo2000 and finagolfin as code owners March 14, 2025 08:22

robertkirkman force-pushed the exclude-pycache-during-extract-into-massagedir branch 5 times, most recently from 84e2f31 to 5d3a7a9 Compare March 15, 2025 10:44

TomJo2000 approved these changes Mar 16, 2025

View reviewed changes

robertkirkman marked this pull request as draft March 16, 2025 02:54

robertkirkman force-pushed the exclude-pycache-during-extract-into-massagedir branch from 5d3a7a9 to cd049e3 Compare March 16, 2025 16:07

robertkirkman marked this pull request as ready for review March 16, 2025 17:47

Maxython force-pushed the exclude-pycache-during-extract-into-massagedir branch from 9c780b7 to e7253ee Compare March 17, 2025 12:17

robertkirkman marked this pull request as draft March 17, 2025 19:19

Maxython force-pushed the exclude-pycache-during-extract-into-massagedir branch 3 times, most recently from 5b68174 to 9266459 Compare March 18, 2025 10:12

Maxython force-pushed the exclude-pycache-during-extract-into-massagedir branch 2 times, most recently from f1ec5dd to 8331f6f Compare April 1, 2025 08:13

robertkirkman force-pushed the exclude-pycache-during-extract-into-massagedir branch 2 times, most recently from c3ea7e4 to 5dbdaf6 Compare October 18, 2025 22:27

robertkirkman force-pushed the exclude-pycache-during-extract-into-massagedir branch from 5dbdaf6 to 37a652a Compare October 19, 2025 16:38

robertkirkman force-pushed the exclude-pycache-during-extract-into-massagedir branch 4 times, most recently from 2213234 to bc68ffc Compare October 25, 2025 16:29

robertkirkman force-pushed the exclude-pycache-during-extract-into-massagedir branch 6 times, most recently from a0852ca to 660a198 Compare November 2, 2025 06:54

robertkirkman force-pushed the exclude-pycache-during-extract-into-massagedir branch 2 times, most recently from 0d6bb35 to d3cf364 Compare November 4, 2025 14:32

robertkirkman mentioned this pull request Nov 5, 2025

fix(scripts/setup-termux): add some more packages to continue to synchronize setup-termux.sh closer to setup-ubuntu.sh #27116

Merged

robertkirkman force-pushed the exclude-pycache-during-extract-into-massagedir branch 3 times, most recently from 9761ccb to a90eb2a Compare November 10, 2025 16:52

robertkirkman force-pushed the exclude-pycache-during-extract-into-massagedir branch 2 times, most recently from 169a64c to 502a00e Compare November 17, 2025 21:17

robertkirkman and others added 4 commits November 17, 2025 20:33

robertkirkman debpython temporary commit 3

662145b

robertkirkman debpython temporary commit 4

7a0cf44

robertkirkman debpython temporary commit 5: cinnamon update

43118ec

robertkirkman force-pushed the exclude-pycache-during-extract-into-massagedir branch from 502a00e to 43118ec Compare November 18, 2025 02:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

tree-wide: port debpython to termux #23652

tree-wide: port debpython to termux #23652

robertkirkman commented Mar 7, 2025 •

edited

Loading

Uh oh!

licy183 commented Mar 7, 2025

Uh oh!

robertkirkman commented Mar 8, 2025 •

edited

Loading

Uh oh!

licy183 commented Mar 8, 2025

Uh oh!

robertkirkman commented Mar 14, 2025

Uh oh!

robertkirkman commented Mar 15, 2025 •

edited

Loading

Uh oh!

TomJo2000 left a comment

Uh oh!

robertkirkman commented Mar 16, 2025 •

edited

Loading

Uh oh!

robertkirkman commented Oct 18, 2025

Uh oh!

TomJo2000 commented Oct 19, 2025

Uh oh!

robertkirkman commented Oct 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

tree-wide: port debpython to termux #23652

Are you sure you want to change the base?

tree-wide: port debpython to termux #23652

Conversation

robertkirkman commented Mar 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What is debpython?

How to add a new Python package after this?

Uh oh!

licy183 commented Mar 7, 2025

Uh oh!

robertkirkman commented Mar 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

licy183 commented Mar 8, 2025

Uh oh!

robertkirkman commented Mar 14, 2025

Uh oh!

robertkirkman commented Mar 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

TomJo2000 left a comment

Choose a reason for hiding this comment

Uh oh!

robertkirkman commented Mar 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

robertkirkman commented Oct 18, 2025

Uh oh!

TomJo2000 commented Oct 19, 2025

Uh oh!

robertkirkman commented Oct 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

robertkirkman commented Mar 7, 2025 •

edited

Loading

robertkirkman commented Mar 8, 2025 •

edited

Loading

robertkirkman commented Mar 15, 2025 •

edited

Loading

robertkirkman commented Mar 16, 2025 •

edited

Loading