I had considered this in

• tree-wide: port debpython to termux  #23652

but, licy183 said:

In Debain, they have a postinst script to compile the modules when new packages are installed, and a prerm script to clean up __pycache__ dir and *.pyc files. I have done the same cleanup logic in pypy3 on Termux.

See /usr/share/python3/runtime.d/public_modules.rt{install,remove} in Debain.

Because of that, next I was going to do that instead,

https://salsa.debian.org/cpython-team/python3-defaults/-/blob/5348f704668c0b6c360b6c6fb10153b9c2898af5/debian/python3.postinst.in#L48

https://salsa.debian.org/cpython-team/python3-defaults/-/blob/5348f704668c0b6c360b6c6fb10153b9c2898af5/debian/python3.prerm#L5

https://salsa.debian.org/cpython-team/python3-defaults/-/blob/5348f704668c0b6c360b6c6fb10153b9c2898af5/py3compile

https://salsa.debian.org/cpython-team/python3-defaults/-/blob/5348f704668c0b6c360b6c6fb10153b9c2898af5/py3clean

https://salsa.debian.org/cpython-team/python3-defaults/-/blob/5348f704668c0b6c360b6c6fb10153b9c2898af5/runtime.d/public_modules.rtinstall

https://salsa.debian.org/cpython-team/python3-defaults/-/blob/5348f704668c0b6c360b6c6fb10153b9c2898af5/runtime.d/public_modules.rtremove

do you think it would be better to try to do that?

Never mind, I did it and while doing so, I realized that those things are kind of not compatible with pacman, only apt/dpkg, so I made sure to turn them all off for pacman.

I've reviewed your PR and have some questions, but I'll ask them in the PR. As for my PR, here's some context for what I'm trying to solve: with recent changes to the way PATH is configured, glibc packages are compiled using commands from glibc packages (i.e. from /data/data/com.termux/files/usr/glibc/bin) including python (if installed). Because of this, the resulting packages are starting to contain __pycache__ directories, which in turn create a conflict when installing these packages. I'm trying to get rid of these __pycache__ directories, but not get rid of all __pycache__ directories since (if I understand correctly) some of them are important (e.g. in python).

glibc packages are compiled using commands from glibc packages (i.e. from /data/data/com.termux/files/usr/glibc/bin) including python (if installed). Because of this, the resulting packages are starting to contain pycache directories, which in turn create a conflict when installing these packages. I'm trying to get rid of these pycache directories, but not get rid of all pycache directories

That is very interesting - it seems like the problem is very comparable to the problem that my PR is solving, but with the opposite $TERMUX_PACKAGE_LIBRARY.

If I replace some words, this description turns into what my PR is solving:

$TERMUX_ON_DEVICE_BUILD=true packages are compiled using commands from bionic packages (i.e. from /data/data/com.termux/files/usr/bin) including python (if installed). Because of this, the resulting packages are containing __pycache__ directories, which in turn create a conflict when installing these packages. I'm trying to get rid of these __pycache__ directories, but exactly copy how the distro Debian does it by managing the __pycache__ directories at install-time, not build-time

(if I understand correctly) some of them are important (e.g. in python).

When I tested, it seemed to me like they are all technically optional, this is exemplified by the distro Debian, which does not actually store any __pycache__ directories inside any of its .deb files as far as I could find, instead it generates them during the installation of the .deb file using the py3compile command. My PR creates a Termux-compatible fork of the py3compile command and calls it from the postinst script of every package that contains .py files in $TERMUX_PREFIX/lib/python$TERMUX_PYTHON_VERSION.

https://packages.debian.org/sid/amd64/libpython3.12-minimal/filelist

If I replace some words, this description turns into what my PR is solving:

$TERMUX_ON_DEVICE_BUILD=true packages are compiled using commands from bionic packages (i.e. from /data/data/com.termux/files/usr/bin) including python (if installed). Because of this, the resulting packages are containing pycache directories, which in turn create a conflict when installing these packages. I'm trying to get rid of these pycache directories, but exactly copy how the distro Debian does it by managing the pycache directories at install-time, not build-time

I would like to note that in the case of compiling glibc packages, this happens on the device (i.e. Termux) and on the docker environment.

When I tested, it seemed to me like they are all technically optional, this is exemplified by the distro Debian, which does not actually store any pycache directories inside any of its .deb files as far as I could find, instead it generates them during the installation of the .deb file using the py3compile command. My PR creates a Termux-compatible fork of the py3compile command and calls it from the postinst script of every package that contains .py files in $TERMUX_PREFIX/lib/python$TERMUX_PYTHON_VERSION.

The idea of ​​the solution is interesting, but your implementation of this idea in the PR confuses me in the following points:

1. you noted that this implementation is not compatible with pacman, and I do not understand why (I need to study this).
2. it seems to me that the activation of termux_create_python_debscripts can be made automatic in scripts, so as not to run it manually each time in build.sh scripts.

1. you noted that this implementation is not compatible with pacman, and I do not understand why (I need to study this).

I think that maybe it can work with pacman, with some additional work, but basically does not right now, so I just refrained from claiming it does for now.

the reason, or at least the first reason that stood out to me, is because it is internally calling the dpkg -L command. As we know, there is a pacman command that is very similar, pacman -Ql, so it might be possible to make it use and correctly parse pacman -Ql if dpkg is not found, but I just have not tested that yet.

2. it seems to me that the activation of termux_create_python_debscripts can be made automatic in scripts, so as not to run it manually each time in build.sh scripts.

Yes, it seems to me like that as well, and in fact, debian does this using its debhelper module called dh-python and that is how Debian does that part, but I did not yet for the following reasons:

1. I noticed that many packages have a preexisting termux_step_create_debscripts() already that does a different repetitive action, which is pip3 install $TERMUX_PKG_PYTHON_TARGET_DEPS. It occurred to me that maybe I could also combine all of those into a single instance in the scripts folder that first tests whether $TERMUX_PKG_PYTHON_TARGET_DEPS is non-empty, and if so, then automatically inserts an instance of pip3 install $TERMUX_PKG_PYTHON_TARGET_DEPS into the postinst script. However, I do not know why that was not already the case - because I saw so many of these, I thought "maybe there is a reason why the postinst and prerm scripts are supposed to be created only inside build.sh files, not inside the scripts folder" - so I followed the preexisting pattern exactly without deviating for now.

2. when I saw that the python package had an instance of exit 0 preexisting on the end of its postinst script, I became concerned because, I thought that if there were other cases of that, it would mean that I cannot make termux_create_python_debscripts() simply check for already-existing instances of postinst and append to the end of the postinst script if already present, because the stray exit 0 would come in the middle of the file and terminate the script before it reaches my code. I haven't actually seen any other cases of that though, as far as I noticed, so if it is ok to remove the exit 0 from this line, then it might make more intense refactoring possible.

Do you know why the pip3 install ${TERMUX_PKG_PYTHON_TARGET_DEPS} lines are not already consolidated into a single instance, and if it would be OK to do that, do you think I could just try refactoring all of them into a single global check simultaneously with all of the instances of termux_create_python_debscripts()?

In order to globally refactor termux_create_python_debscripts(), I just would need to find a reliable and concise way to detect whether the package is going to install a .py file anywhere into $TERMUX_PREFIX/lib/python$TERMUX_PYTHON_VERSION/ or not. If that is possible, then I can make the global function autogenerate the correct postinst and prerm scripts without mistakes.
Edit: now I have made that and put it in my PR