这是indexloc提供的服务,不要输入任何密码
Skip to content

bump(scripts/build/setup/termux_setup_python_pip): setuptools 78.1.0, wheel 0.46.1 #24213

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 13, 2025

Conversation

robertkirkman
Copy link
Contributor

@robertkirkman robertkirkman commented Apr 9, 2025

  • wheel version 0.46+ now requires setuptools version 70 or newer

  • Lock wheel version at 0.46.1 so that if a future wheel update causes a similar problem in the future, it doesn't immediately propagate into termux-packages without us explicitly bumping it. Recommended by Tomjo2000 here: Libxml2 rebuilds: Part 2 - Small packages #24062 (comment)

  • I have confirmed that this change works to fix the build of python-lxml both

    • in TERMUX_ON_DEVICE_BUILD=false mode (temporarily broken for 1 week),
      • fixes error: invalid command 'bdist_wheel'
    • and also in TERMUX_ON_DEVICE_BUILD=true mode (broken for 6 months).
      • fixes ModuleNotFoundError: No module named 'setuptools'
  • It seems appropriate to remove the if [ "${TERMUX_PYTHON_VERSION#*.}" -lt "12" ] block because there is no longer any python3.11 so it does not seem like that code is reachable anymore.

  • python-torchvision needed ajustment of its custom CFLAGS to be CXXFLAGS instead, because the newer setuptools is now compiling it using aarch64-linux-android-clang++ instead of aarch64-linux-android-clang.

  • python-pip and python-pillow both had locked setuptools in their TERMUX_PKG_PYTHON_COMMON_DEPS at versions slightly newer than the old global version, now that the global version is newer than either of those, they do not seem to need that anymore.

Copy link
Member

@licy183 licy183 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure whether this will break the building of other packages. It is better to test other packages too.

@robertkirkman robertkirkman marked this pull request as draft April 9, 2025 04:27
@robertkirkman
Copy link
Contributor Author

I'm not sure whether this will break the building of other packages. It is better to test other packages too.

Ok I will try to find all packages which depend on setuptools being installed in the pip virtualenv, and check them also.

@robertkirkman robertkirkman force-pushed the setuptools-78.1.0 branch 2 times, most recently from 6418d77 to 3f411f0 Compare April 9, 2025 09:47
@robertkirkman
Copy link
Contributor Author

robertkirkman commented Apr 9, 2025

I am not sure if I can find every single possible package that uses termux_setup_python_pip(), but I have run a build of a relatively large number of them using this command, while this PR is applied.

scripts/run-docker.sh ./build-package.sh -I -f calcurse electrum hash-slinger lv2 matplotlib nala python-brotli python-cryptography python-lxml python-pycryptodomex python-tflite-runtime python-tldp python-torch python-torchaudio python-torchvision python-xlib python-yt-dlp rdircd srt2vobsub termux-gui-package termux-gui-pm units ytui-music inkscape orca pyqt5 python-scipy python-pynvim python-pillow seafile-client speechd meld pyatspi python-pip

Of all of them, there was one failure in python-torchvision, that is caused by bumping setuptools to version 78.1.0.

python-torchvision now (after bumping setuptools but before compatibility patch)

aarch64-linux-android-clang++ -fstack-protector-strong -Oz -DUSE_PYTHON -isystem/data/data/com.termux/files/usr/include/c++/v1 -isystem/data/data/com.termux/files/usr/include -fPIC -I/home/builder/.termux-build/python-torchvision/src/torchvision/csrc -I/home/builder/.termux-build/python3.12-crossenv-prefix-bionic-aarch64/cross/include -I/data/data/com.termux/files/usr/include/python3.12 -c /home/builder/.termux-build/python-torchvision/src/torchvision/csrc/ops/autocast/deform_conv2d_kernel.cpp -o build/temp.linux-aarch64-cpython-312/home/builder/.termux-build/python-torchvision/src/torchvision/csrc/ops/autocast/deform_conv2d_kernel.o -g0
In file included from /home/builder/.termux-build/python-torchvision/src/torchvision/csrc/ops/autocast/deform_conv2d_kernel.cpp:1:
/home/builder/.termux-build/python-torchvision/src/torchvision/csrc/ops/autocast/../deform_conv2d.h:3:10: fatal error: 'ATen/ATen.h' file not found
      3 | #include <ATen/ATen.h>
        |          ^~~~~~~~~~~~~
1 error generated.

It seems like it happened because previously, the old setuptools was, for some reason, compiling the .cpp files of python-torchvision using aarch64-linux-android-clang instead of aarch64-linux-android-clang++ (I understand that is possible for build systems to do, but it is weird. Generally, clang is intended mainly for .c files and clang++ is intended mainly for .cpp files).

python-torchvision previously (was working)

aarch64-linux-android-clang -fno-strict-overflow -Wsign-compare -Wunreachable-code -DNDEBUG -g -O3 -Wall -fstack-protector-strong -O3 -fstack-protector-strong -O3 -fstack-protector-strong -Oz -I/data/data/com.termux/files/usr/lib/python3.12/site-packages/torch/include -I/data/data/com.termux/files/usr/lib/python3.12/site-packages/torch/include/torch/csrc/api/include -isystem/data/data/com.termux/files/usr/include/c++/v1 -isystem/data/data/com.termux/files/usr/include -fPIC -I/home/builder/.termux-build/python-torchvision/src/torchvision/csrc -I/home/builder/.termux-build/python3.12-crossenv-prefix-bionic-aarch64/cross/include -I/data/data/com.termux/files/usr/include/python3.12 -c /home/builder/.termux-build/python-torchvision/src/torchvision/csrc/ops/autocast/deform_conv2d_kernel.cpp -o build/temp.linux-aarch64-cpython-312/home/builder/.termux-build/python-torchvision/src/torchvision/csrc/ops/autocast/deform_conv2d_kernel.o -g0

The newer setuptools is compiling the .cpp files using aarch64-linux-android-clang++, so the CFLAGS modifications here were not having an effect anymore and so the .cpp file did not get its proper include search directories.

CFLAGS+=" -I${TERMUX_PYTHON_HOME}/site-packages/torch/include"
CFLAGS+=" -I${TERMUX_PYTHON_HOME}/site-packages/torch/include/torch/csrc/api/include"

Simply making this change seems to make the error go away and python-torchvision build successfully with setuptools 78.1.0.

python-torchvision setuptools 78.1.0 compatibility patch

--- a/packages/python-torchvision/build.sh
+++ b/packages/python-torchvision/build.sh
@@ -9,8 +9,8 @@ TERMUX_PKG_DEPENDS="libc++, ffmpeg, python, python-numpy, python-pillow, python-
 TERMUX_PKG_PYTHON_COMMON_DEPS="wheel, setuptools"
 
 termux_step_pre_configure() {
-	CFLAGS+=" -I${TERMUX_PYTHON_HOME}/site-packages/torch/include"
-	CFLAGS+=" -I${TERMUX_PYTHON_HOME}/site-packages/torch/include/torch/csrc/api/include"
+	CXXFLAGS+=" -I${TERMUX_PYTHON_HOME}/site-packages/torch/include"
+	CXXFLAGS+=" -I${TERMUX_PYTHON_HOME}/site-packages/torch/include/torch/csrc/api/include"
 	CXXFLAGS+=" -DUSE_PYTHON"
 	LDFLAGS+=" -ltorch_cpu -ltorch_python -lc10"
 

I could not find any other packages that need to apply custom CXXFLAGS or CFLAGS in build.sh to setuptools.


python-pip and python-pillow had explicit setuptools==.., in their TERMUX_PKG_PYTHON_COMMON_DEPS, since the global setuptools will be a higher version than either of those was now, is it now OK to remove the locked versions from those packages? They do not seem to have build problems when I do that.

@robertkirkman robertkirkman marked this pull request as ready for review April 9, 2025 09:57
@robertkirkman
Copy link
Contributor Author

This is a dependency for rebuilding reverse dependencies of libxml2 so that libxml2 can be bumped to 2.14, so I will merge this soon if no other packages that need changes for compatibility with setuptools 78 can be found, and afterward, if a different package begins failing because of setuptools 78, please ping me and I will try to update it to be compatible with setuptools 78.

Copy link
Member

@TomJo2000 TomJo2000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I haven't found any further build issues from this PR in local testing.

I'm not 100% sure I've tested all packages impacted by this though.

@robertkirkman
Copy link
Contributor Author

It turns out that this PR is now somewhat lower priority than it was a few days ago, because wheel 0.46.1 has now been "yanked" on PyPi, which has externally manipulated the termux-packages termux_setup_python_pip() function such that it now only installs wheel 0.45.1 again instead of wheel 0.46.1 like it did a few days ago.

https://pypi.org/project/wheel/#history

Because of that, I can now wait a bit longer before merging this PR, maybe 24 hours or until something else specifically needs it again.

… wheel 0.46.1

- `wheel` version 0.46+ now requires `setuptools` version 70 or newer
  - discussed upstream here: pypa/wheel#660
  - discovered in termux-packages here: termux#24062 (comment)

- Lock `wheel` version at 0.46.1 so that if a future `wheel` update causes a similar problem in the future, it doesn't immediately propagate into termux-packages without us explicitly bumping it. Recommended by Tomjo2000 here: termux#24062 (comment)

- I have confirmed that this change works to fix the build of `python-lxml` **both**
  - **in `TERMUX_ON_DEVICE_BUILD=false` mode (broken temporarily for 1 week)**,
    - fixes `error: invalid command 'bdist_wheel'`
  - **and also in `TERMUX_ON_DEVICE_BUILD=true` mode (broken for 6 months)**.
    - fixes `ModuleNotFoundError: No module named 'setuptools'`

- It seems appropriate to remove the `if [ "${TERMUX_PYTHON_VERSION#*.}" -lt "12" ]` block because there is no longer any `python3.11` so it does not seem like that code is reachable anymore.

- `python-torchvision` needed ajustment of its custom `CFLAGS` to be `CXXFLAGS` instead, because the newer `setuptools` is now compiling it using `aarch64-linux-android-clang++` instead of `aarch64-linux-android-clang`.

- `python-pip` and `python-pillow` both had locked `setuptools` in their `TERMUX_PKG_PYTHON_COMMON_DEPS` at versions slightly newer than the old global version, now that the global version is newer than either of those, they do not seem to need that anymore.
@robertkirkman robertkirkman changed the title bump(scripts/build/setup/termux_setup_python_pip): setuptools 78.1.0 bump(scripts/build/setup/termux_setup_python_pip): setuptools 78.1.0, wheel 0.46.1 Apr 12, 2025
@robertkirkman
Copy link
Contributor Author

As suggested here I have changed this PR to also lock wheel in the same place where setuptools is being locked, so that they both remain at versions that are compatible with each other and an external PyPi bump doesn't automatically propagate into termux_setup_python_pip() again in the future without us explicitly bumping it.

@robertkirkman
Copy link
Contributor Author

Ok, I have not found any other packages yet that need changes to work with this, so I will merge this 24 hours from now.

@TomJo2000
Copy link
Member

It has now been 24 hours, I'm merging this PR.

@TomJo2000 TomJo2000 merged commit f8f8008 into termux:master Apr 13, 2025
9 checks passed
@licy183
Copy link
Member

licy183 commented May 2, 2025

Should consider reverting this as setuptools>=75.3.2 will rewrite shebang.

See also pypa/setuptools#4934.

@robertkirkman
Copy link
Contributor Author

robertkirkman commented May 2, 2025

Should consider reverting this as setuptools>=75.3.2 will rewrite shebang.

See also pypa/setuptools#4934.

I bisected the problem to 4dbc002, not this PR, but if you believe that both should be reverted, then maybe you are correct, I am not sure.

Shouldn't correct behavior of termux_step_massage(), when fixed, prevent all of this kind of incorrect shebangs? (there are a large number of packages that have shebangs replaced there, not only python-related)

Note, for example, that

completed successfully without encountering this problem, because, it happened before 4dbc002, despite happening after this PR (bump setuptools).

@licy183
Copy link
Member

licy183 commented May 2, 2025

I'm not sure. But it think the shebang should get fixed in massage step. You're right...

@robertkirkman
Copy link
Contributor Author

@licy183 I just simulated reverting this PR (setuptools bump) without reverting 4dbc002, by doing this locally, and at least as far as I can tell, it does not affect the shebang problem in

--- a/scripts/build/setup/termux_setup_python_pip.sh
+++ b/scripts/build/setup/termux_setup_python_pip.sh
@@ -62,8 +62,8 @@ termux_setup_python_pip() {
 
 		# Since 3.12, distutils is removed from python, but setuptools>=60 provides it
 		# Since wheel 0.46, setuptools>=70 is required to provide bdist_wheel
-		build-pip install 'setuptools==78.1.0' 'wheel==0.46.1'
-		cross-pip install 'setuptools==78.1.0' 'wheel==0.46.1'
+		build-pip install 'setuptools==69.5.1' 'wheel==0.45.1'
+		cross-pip install 'setuptools==69.5.1' 'wheel==0.45.1'
 
 		export PATH="${TERMUX_PYTHON_CROSSENV_PREFIX}/build/bin:${PATH}"
 		local _CROSS_PATH="${TERMUX_PYTHON_CROSSENV_PREFIX}/cross/bin"

Rebuilding and reinstalling python-pip with this change only, does not seem to prevent the error pip: cannot execute: required file not found, and the shebang remains #!/home/builder/.termux-build/python3.12-crossenv-prefix-bionic-aarch64/cross/bin/python3.12.

Only changes that consider the bisection point as 4dbc002, seem to be able to impact this bug.

However, I could be wrong, if you think I have missed some part of this test, you can let me know.

robertkirkman added a commit to robertkirkman/termux-packages that referenced this pull request Jul 2, 2025
…LAGS` and `CXXFLAGS`

- During termux#24213,
  `python-torchvision` would not build unless the include paths were
  added to `CXXFLAGS`, but now the package is failing to build again
  unless the include paths are added to `CFLAGS` also, so add the same
  include paths to both `CFLAGS` and `CXXFLAGS` to cover both cases.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants