这是indexloc提供的服务,不要输入任何密码
Skip to content

Conversation

@twaik
Copy link
Member

@twaik twaik commented Feb 27, 2025

Fixes on-device builds.

@twaik
Copy link
Member Author

twaik commented Feb 27, 2025

Works fine on my device.

@twaik twaik force-pushed the fix-on-device-build-cleanup branch from 936093e to 0a5c961 Compare February 27, 2025 17:12
Copy link
Member

@robertkirkman robertkirkman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok now it is working on my device as expected.

@twaik twaik force-pushed the fix-on-device-build-cleanup branch 2 times, most recently from e76ec14 to edbbbf5 Compare February 27, 2025 17:27
@twaik twaik force-pushed the fix-on-device-build-cleanup branch 4 times, most recently from 330b717 to d78d634 Compare February 27, 2025 18:19
@twaik twaik requested a review from TomJo2000 February 27, 2025 19:23
@twaik
Copy link
Member Author

twaik commented Feb 27, 2025

Since this PR makes newly added build step disabled by default it should be harmless.
I will merge it in 12 hours if there are no objections.

# Get available space (df outputs sizes in 1024-byte blocks)
local space_available="$(df -P "/var/lib/docker" | awk 'NR==2 {print $4}')"
# Get available space in bytes
local space_available="$(df "/var/lib/docker" | awk 'NR==2 {print $4 * 1024}')"
Copy link
Contributor

@truboxl truboxl Feb 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a case if /var/lib/docker not exist or the command fails, fallback to check current location disk space

build-package.sh Outdated
fi
;;
-c) TERMUX_CONTINUE_BUILD=true;;
-C) TERMUX_CLEANUP_PACKAGES=true;;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rename to TERMUX_CLEANUP_BUILT_PACKAGES_ON_LOW_DISK_SPACE

@twaik twaik force-pushed the fix-on-device-build-cleanup branch from d78d634 to 833f12b Compare February 28, 2025 10:20
@agnostic-apollo
Copy link
Member

local TERMUX_PACKAGES_DIRECTORIES="$(jq --raw-output 'del(.pkg_format) | keys | .[]' "${TERMUX_SCRIPTDIR}"/repo.json)"
# Build package name regex to be used with `find`, avoiding loops.
local PKG_REGEX="$(find ${TERMUX_PACKAGES_DIRECTORIES} -mindepth 1 -maxdepth 1 -type d -printf '%f|')^$"
# Exclude current package from the list.
PKG_REGEX="${PKG_REGEX//$TERMUX_PKG_NAME|/}"
echo "INFO: cleaning up some disk space for building \"${TERMUX_PKG_NAME}\"."
find "$TERMUX_TOPDIR" -mindepth 1 -maxdepth 1 -type d -regextype posix-extended -regex ".*/($PKG_REGEX)$" -exec rm -rf {} +

  1. Using local and sub shell together hides the result value of the command and set -e abort won't happen. All local variables must be declared beforehand.
$ func() { set -e; local res="$(false)"; echo here1; }
(func;)
func1() { set -e; local res; res="$(false)"; echo here2; }
(func1;)
here1
  1. Using -printf '%f|' will result in last file to also have | appended, which will result in a dangling | in the regex, which is invalid.
  2. Package names are not safe to be parsed as regex, for example, libc++, libsigc++-3.0, etc have regex special characters like . and +, which can have unexpected consequences. Additionally, a double ++ would be invalid too.
  3. Instead of using .* in .*/($PKG_REGEX)$", cd first and use ^\./.
local packages packages_regex

packages="$(find ${TERMUX_PACKAGES_DIRECTORIES} -mindepth 1 -maxdepth 1 -type d -printf '%f\n')"
[[ -z "$packages" ]] && return 0

# Exclude current package.
packages="$(printf "%s" "$packages" | grep -Fxv "$TERMUX_PKG_NAME")"
[[ -z "$packages" ]] && return 0

packages_regex="$(printf "%s" "$packages" | sed -zE 's/[][\.|$(){}?+*^]/\\&/g' | sed -E 's/(.*)/(\1)/g' | sed -zE -e 's/[\n]+/|/g' -e 's/(.*)/(\1)/g')"

(cd "$TERMUX_TOPDIR" && find . -mindepth 1 -maxdepth 1 -type d -regextype posix-extended -regex "^\./$packages_regex$" -exec rm -rf "{}" +)

@twaik
Copy link
Member Author

twaik commented Mar 2, 2025

1. Using local and sub shell together hides the result value of the command and set -e abort won't happen. All local variables must be declared beforehand.

In this case nothing bad will happen, cleanup will not be performed. Seems to be harmless.

2. Using -printf '%f|' will result in last file to also have | appended, which will result in a dangling | in the regex, which is invalid.

Correct, and that is the reason for ^$ dummy token to be added. Empty filenames are not possible in our case so it will have no matches.
Also | is not a part of any packagename in termux-packages so IMHO it should not be escaped and can be safely used here.

3. Package names are not safe to be parsed as regex, for example, libc++, libsigc++-3.0, etc have regex special characters like . and +, which can have unexpected consequences. Additionally, a double ++ would be invalid too.

Correct. Did not think about it.

4. Instead of using .* in .*/($PKG_REGEX)$", cd first and use ^\./.

find appends a string literal passed to commandline as a part of response so it should be safe to use a response without cd, isn't it?

@twaik twaik force-pushed the fix-on-device-build-cleanup branch from 8be3236 to 833f12b Compare March 2, 2025 05:48
@twaik twaik requested a review from agnostic-apollo March 3, 2025 05:04
@twaik twaik force-pushed the fix-on-device-build-cleanup branch from 833f12b to 0e26cf6 Compare March 3, 2025 05:18
@twaik twaik requested a review from truboxl March 3, 2025 05:24
@twaik twaik force-pushed the fix-on-device-build-cleanup branch from 0e26cf6 to 724ead3 Compare March 3, 2025 06:29
@truboxl
Copy link
Contributor

truboxl commented Mar 3, 2025

Please init TERMUX_CLEANUP_BUILT_PACKAGES_ON_LOW_DISK_SPACE variable with false value in scripts/build/termux_step_setup_variables.sh instead

to allow reuse outside scripts/build/termux_step_cleanup_packages.sh

@robertkirkman
Copy link
Member

Truboxl, I asked about that but, twaik explained that there is some reason it is better to stay this way here

@truboxl
Copy link
Contributor

truboxl commented Mar 3, 2025

It would be extremely helpful and speed up review process of changing the build scripts by having CI for on device build. Not that we are allowing on device built packages to be uploaded to server yet (with some exceptions like pypy).

@twaik
Copy link
Member Author

twaik commented Mar 3, 2025

It would be extremely helpful and speed up review process of changing the build scripts by having CI for on device build. Not that we are allowing on device built packages to be uploaded to server yet (with some exceptions like pypy).

Probably we should create termux-docker-based workflow for checking build-system capabilities for all cases (host, docker and termux-docker (aka on-device)).

Copy link
Member

@TomJo2000 TomJo2000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I'd just like the space threshold to be standardized.

@twaik twaik force-pushed the fix-on-device-build-cleanup branch from 724ead3 to 02f96bd Compare March 3, 2025 15:11
@twaik
Copy link
Member Author

twaik commented Mar 3, 2025

Seems to be fine, I'll merge this in 12 hours if no one objects.

@robertkirkman
Copy link
Member

robertkirkman commented Mar 3, 2025

It would be extremely helpful and speed up review process of changing the build scripts by having CI for on device build. Not that we are allowing on device built packages to be uploaded to server yet (with some exceptions like pypy).

I discussed some information about that here, i believe licy183 might try to make that, but I will try to make it also.

I created a $TERMUX_ON_DEVICE_BUILD=true-compatible build.sh for pypy as a test and it works, but I will wait to upload it until after there is a workflow capable of testing it in CI.

There are a lot of packages that currently have various errors when built with $TERMUX_ON_DEVICE_BUILD=true, so if such a workflow is enabled, it would fail to build a lot of packages, but maybe it would be good if it helps to fix those errors.

Probably we should create termux-docker-based workflow for checking build-system capabilities for all cases (host, docker and termux-docker (aka on-device)).

There are some packages that rely on audio or graphics libraries, which I believe termux-docker cannot currently build or test effectively with exactly accurate behavior to what a full AVD would, because of not having, for one example, OpenSL ES libraries or stubs yet, so any workflow doing this would need to be limited primarily to purely "headless" CLI packages first, but it would still be a good start.

@agnostic-apollo
Copy link
Member

In this case nothing bad will happen, cleanup will not be performed. Seems to be harmless.

In your old code, you would be getting an empty variable for packages when command fails, and then using it to generate a regex and then using it to remove directories. If something bad doesn't happen, it wouldn't be because its harmless, but because you were lucky.

You still haven't separated the local declaration from assignment with commands. Fix it.

Correct, and that is the reason for ^$ dummy token to be added. Empty filenames are not possible in our case so it will have no matches.

You would still not be creating a valid or at least logical regex. That dummy token will be added to ".*/($PKG_REGEX)$" and you will get something like ".*/(foo|bar|^$)$", where there are double end anchors $.

find appends a string literal passed to commandline as a part of response so it should be safe to use a response without cd, isn't it?

Yes, it should be fine to do that as well, at the cost of slight cost if there are lot of directories.

@twaik
Copy link
Member Author

twaik commented Mar 4, 2025

You still haven't separated the local declaration from assignment with commands. Fix it.

You mean declare all local variables on separate line at the start of the function?

You would still not be creating a valid or at least logical regex. That dummy token will be added to ".*/($PKG_REGEX)$" and you will get something like ".*/(foo|bar|^$)$", where there are double end anchors $.

The whole point was to create a token that will not match in any case :) .

@agnostic-apollo
Copy link
Member

@twaik
Copy link
Member Author

twaik commented Mar 4, 2025

Oh, ok. TIL.

@twaik twaik force-pushed the fix-on-device-build-cleanup branch from 02f96bd to dc6ea3a Compare March 4, 2025 05:06
@twaik
Copy link
Member Author

twaik commented Mar 4, 2025

Done.

@twaik
Copy link
Member Author

twaik commented Mar 4, 2025

So now it should be fine, right?

@agnostic-apollo
Copy link
Member

Yes, thanks

@twaik twaik merged commit ff5c2d6 into master Mar 4, 2025
@twaik twaik deleted the fix-on-device-build-cleanup branch March 4, 2025 05:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants