tree-wide: support full bootstrapping of ffplay, rust, and mesa
#24669
Conversation
|
new @Maxython I am just interested to know whether you are OK with the changes to |
I approve |
packages/python-brotli/build.sh
Outdated
There was a problem hiding this comment.
Is there a specific reason we're bringing python-brotli back as a standalone package?
I thought we specifically made it a subpackage of brotli earlier?
There was a problem hiding this comment.
Well, it is because that PR causes this command to fail (note absence of -I):
scripts/run-docker.sh ./build-package.sh python
with this error:
ERROR: /data/data/com.termux/files/usr/bin/python3.12 does not exist
There was a problem hiding this comment.
Interesting.
No idea why it would do that.
Did we create an accidental dependency cycle with that?
E.g does python transiently depend on brotli through one of its dependencies,
but brotli depended on python after #24397?
There was a problem hiding this comment.
I wrote down the dependency cycle as it happened at the time hidden somewhere in one of the extremely large text walls in
Here it is copied and pasted from there:
This happened because
pythonhas a build dependency ontk, andtkhas a build dependency onfontconfig, andfontconfighas a build dependency onfreetype, andfreetypehas a build dependency onbrotli, andpython-brotlihas a build dependency onpython-pip, which becomes a build dependency ofbrotliafter
and
python-piphas a build dependency onpython, so this creates a cyclical dependency error when bootstrapping. This can be worked around by reverting #24397
There was a problem hiding this comment.
Since the time I posted that, several other dependencies have shifted slightly, which might cause the dependency cycle to take a slightly different route if that were tested again today, but it is believed that the same or a similar error would still occur, and that reverting the PR is still necessary in order to successfully run scripts/run-docker.sh ./build-package.sh python.
| # always remove this marker because this package is built in termux_step_host_build() | ||
| # this prevents "ERROR: No files in package." when the package is built again without deleting | ||
| # the docker container. | ||
| rm -rf $TERMUX_HOSTBUILD_MARKER |
There was a problem hiding this comment.
First time seeing that variable.
Is this documented?
There was a problem hiding this comment.
There are several other places in the repository where this is done in order to force termux_step_host_build() to always run on every build and never cache. Here is one of them:
termux-packages/x11-packages/virglrenderer-android/build.sh
Lines 72 to 75 in d26858e
reasons this can be necessary vary between each package.
There was a problem hiding this comment.
Actually, while on the subject, I just noticed that technically glib requires this too, because if it doesn't have it, then this command:
scripts/run-docker.sh ./build-package.sh -a all gtk3
will fail with this error:
.../files/usr/bin/glib-compile-resources: Exec format error
(because glib-cross does not get properly installed into $TERMUX_PREFIX if an operation that deletes $TERMUX_PREFIX without deleting all hostbuild markers, like -a all, is combined with the mode that runs when -I is not used)
Basically any package that installs something into $TERMUX_PREFIX during termux_step_host_build() requires it. I'm doing them one by one while they're touched by the dependency tree of the packages mentioned in each PR related to this subject, I guess.
There was a problem hiding this comment.
I guess it would be possible to add a check that counts files modified in $TERMUX_PREFIX since termux_step_create_timestamp_file() right after the end of termux_step_host_build() but before anything else runs, and remove/don't create $TERMUX_HOSTBUILD_MARKER if that number of files is greater than 0, and that would fix some of these automatically, but it wouldn't cover all of them and that might be kind of an overly invasive change.
|
It says @codingWiz-rick @hansm629 if you want to test it, please try testing this version from this floating commit if you can, https://github.com/termux/termux-packages/commits/333c3a3bf60952952b9561c41f8e71432361b95c/ until after both parts are merged. because it is too much stuff to fit in one PR, so I have had to remove about half the stuff, and the other half depends on this part, so I will have to put it in the other PR after this one is done. |
333c3a3 to
0980e38
Compare
28c3747 to
6d4e293
Compare
|
When I added even just one more package after removing all reverse dependencies of |
4987082 to
1ffcc8d
Compare
|
The reason it is now building again is because I carefully double-checked and tested my changes to |
I don't think we have a hard and fast "must be this big to be in |
1ffcc8d to
0128fd4
Compare
|
OP continued to expand the scope of the original issue, and while checking that I saw the To limit scope creep, I am drawing a hard line at any changes to the |
|
Hmm, It said This file was last primarily edited in however, at the time when I tested that PR locally, I actually built almost all the same packages that are in this PR locally, and that error did not happen. I will try to reproduce this locally and try to figure out what happened to |
…ny of the function when `$TERMUX_INSTALL_DEPS` is `false` - but skip all of the function when `$TERMUX_PACKAGE_LIBRARY` is `glibc` - recommended long ago by @Maxython, but not organized out into a new PR until today - fixes `ERROR: Failed running '/data/data/com.termux/files/usr/bin/llvm-config', binary or interpreter not executable` during `$TERMUX_ON_DEVICE_BUILD=false` mode when `-I` is _not_ passed to `build-package.sh`, in all reverse dependencies of `llvm-config`, like `mesa` and `rust`.
- Reverts termux#24397 because unfortunately, it causes the error `ERROR: /data/data/com.termux/files/usr/bin/python3.12 does not exist` when the command `scripts/run-docker.sh ./build-package.sh python` (**without `-I`**) is used
… `termux_step_post_make_install()`, not `termux_step_post_massage()` - `termux_step_post_massage()` operates on `$TERMUX_PKG_MASSAGEDIR/$TERMUX_PREFIX`, not `$TERMUX_PREFIX` directly, so previously, `$TERMUX_PREFIX/lib/libSDL2.so` would fail to exist during the use of `build-package.sh` **without `-I`** on reverse dependencies of `sdl2`. - The error this fixes in reverse dependencies looks like this: ``` Hint: ERROR: sdl2 requested but not found ```
…ttempt to detect `vapigen` - This fixes the build of `librsvg` if `scripts/run-docker.sh ./build-package.sh ffplay rust mesa` **without `-I`** was previously used in the same container before building `librsvg`
…essfully if various things were previously built in the same `$TERMUX_PREFIX` - Fixes `mv: cannot overwrite '/data/data/com.termux/files/usr/share/cmake': Directory not empty` if anything else that also has a `$TERMUX_PREFIX/share/cmake` folder was already installed previously - Prevents any similar error from occurring for the `$TERMUX_PREFIX/lib/clang/$llvm_major_version/lib` folder - Fix repeated builds by removing `$TERMUX_HOSTBUILD_MARKER` during `termux_step_configure()`
- `libtheora` 1.2.0 does not search `$TERMUX_PREFIX` for `rm`, `sed`, `sort`, or `cat`
- `libtheora` has not been bumped in many years, and this update **heavily changes the names of the `.so` files** (example: `libtheoradec.so.1.1.4`->`libtheoradec.so`) so all `libtheora` reverse dependencies must be rebuilt.
- The error this fixes during cross-compilation `$TERMUX_ON_DEVICE_BUILD=false` mode while `coreutils` was already installed in the same `$TERMUX_PREFIX` before building, looks like this:
```
/home/builder/.termux-build/libtheora/src/configure: line 13310: /data/data/com.termux/files/usr/bin/rm: cannot execute binary file: Exec format error
/home/builder/.termux-build/libtheora/src/configure: line 13319: /data/data/com.termux/files/usr/bin/sed: cannot execute binary file: Exec format error
/home/builder/.termux-build/libtheora/src/configure: line 13320: ${+set}: bad substitution
/home/builder/.termux-build/libtheora/src/configure: line 6: /data/data/com.termux/files/usr/bin/cat: cannot execute binary file: Exec format error
/home/builder/.termux-build/libtheora/src/configure: line 14: /data/data/com.termux/files/usr/bin/sed: cannot execute binary file: Exec format error
/home/builder/.termux-build/libtheora/src/configure: line 40: /data/data/com.termux/files/usr/bin/sort: cannot execute binary file: Exec format error
/home/builder/.termux-build/libtheora/src/configure: line 37: /data/data/com.termux/files/usr/bin/sed: cannot execute binary file: Exec format error
/home/builder/.termux-build/libtheora/src/configure: line 44: /data/data/com.termux/files/usr/bin/cat: cannot execute binary file: Exec format error
/home/builder/.termux-build/libtheora/src/configure: line 57: /data/data/com.termux/files/usr/bin/sort: cannot execute binary file: Exec format error
/home/builder/.termux-build/libtheora/src/configure: line 79: /data/data/com.termux/files/usr/bin/cat: cannot execute binary file: Exec format error
/home/builder/.termux-build/libtheora/src/configure: line 85: /data/data/com.termux/files/usr/bin/cat: cannot execute binary file: Exec format error
/home/builder/.termux-build/libtheora/src/configure: line 92: /data/data/com.termux/files/usr/bin/rm: cannot execute binary file: Exec format error
```
- Use `--disable-asm` for 32-bit ARM only, because the assembly optimizations for x86 architectures work, but the assembly optimizations for 32-bit ARM fail to build, and were not previously enabled by configure, but are now being enabled automatically.
- Prevents the error `.../files/usr/bin/glib-compile-resources: Exec format error` when the command `scripts/run-docker.sh -a all gtk3` (without `-I`, with `-a all`) is used. - The reason it prevents that is because building without `-I` relies on packages that are formed from files installed into `$TERMUX_PREFIX` during `termux_step_host_build()` like `glib-cross` being built during the build for that architecture, but `-a all` deletes `$TERMUX_PREFIX` in between builds but does not remove hostbuild markers or hostbuild folders, so `glib-cross` will not get actually installed into `$TERMUX_PREFIX` for successive architectures unless its marker is removed.
- `vlc-qt`: fix build with `taglib` 2.0 after termux#24035
- this package is not in x11-repo, so its X11 features should not be automatically enabled even when X11 libraries are detected - fixes the build when X11 libraries are present in `$TERMUX_PREFIX` before the build
- `wasi-libc` takes up about 8 GB of storage space when building, so it is difficult to build it in the same PR with any other package unless it is in `big-pkgs.list`
- Termux's `tor` package is not currently compatible with `libseccomp`, but if `libseccomp` is present in `$TERMUX_PREFIX` before `tor` is built, then this error would happen: `src/lib/sandbox/sandbox.c:890:32: error: use of undeclared identifier 'PF_FILE'`
0128fd4 to
2e3d168
Compare
|
Ok, I rebased it past and the error in CI did not happen now. |
|
I will merge this in 24 hours from now, and these kinds of build errors for any other packages that were not mentioned here will be fixed by other PRs. |
- Fixes #3 - `glib-force-always-hostbuild.patch` was upstreamed in termux/termux-packages#24669
Progress on Some packages cannot be built 3: Revenge of the Xiph #23492
Fixes [Bug]: ffmpeg building #23697
Fixes [Bug]: binary or interpreter not executable. #20336
Possibly fixes (but probably not, not easily reproducible bug) [Bug]: vlc-qt is not working. #24456
Reverts chore(main/brotli): move python-brotli to subpackage #24397
Implements the subset of [RFC] fix $TERMUX_PREFIX pollution #21835 that is specific to
ffplay,rust,mesa, and their entire dependency trees, and not necessarily any other packages other than the other reverse dependencies oflibtheora, which must be rebuilt after the bump. However, the change totorfrom there is included here only because it is, like some other packages here, important for the successful use ofbuild-bootstraps.sh.This PR fixes all errors in the following commands:
packages that need the
brotli-related changes:pythonpython-pippackages that additionally need the
libtheora-related andsdl2-related changes:ffplaypackages that additionally need the
wasi-libc-related andtermux_step_override_config_scripts()-related changes:rustmesaother reverse dependencies of
libtheorathat are revision-bumped and simultaneously cleaned of prefix-pollution from above packages:ffmpeggpacgst-plugins-basemplayervlcgodothandbrakeloverecordmydesktopvlc-qt